Seeping Matter

I hinted at this before, but I’ve started a new open source project called Zoomulus.  It’s a fun little project to product tools and libraries to help us normal people leverage the power of cloud computing more quickly.  I wrote a blog post today explaining why I started the project.

For most, it is just another open source project, but for me, it is a part of me.  There’s not much there yet, but I’m pretty pleased with it anyway.  If you are interested, take a look or even get involved; I’d love to have you onboard.

Yes, you read that correctly.  For far too long all us coding frogs have been neglected and discriminated against.  But no more!  My new software development website, www.codingfrogs.net, is up and running!

I will now take questions.

Q:  What is codingfrogs.net?

A:  It is my new software development blog, dealing with the technical, process, and business aspects of software development.

Q:  Where did you get that outstanding frog picture?

A:  I know, huh!

Q:  No, really – where?

A:  It is the original creation of Rosie Leung, which she made available via Creative Commons on her website.

Q:  Are only coding frogs allowed at that website, or may any type of coding animal visit?

A:  codingfrogs.net welcomes any type of coding animal, but frogs get preferred seating.

Q:  Will you allow other coding frogs to guest-post on codingfrogs.net?

A:  Yes, but only if I let them.

Q:  Isn’t the purpose of this post primarily to drive up your PageRank score for codingfrogs.net, even if only by a tiny amount?

A:  Next question.

Q:  Why codingfrogs.net?

A:  Because the domain name was available, of course.

Q:  Is it true that you created this website to Michelle Barnum would quit complaining about your technical posts?

A:  Pretty much.

The Go Gopher

Earlier this week Google announced their new programming language, called Go.

Usually I don’t get too worked up about programming languages.  I already feel like I know more languages than I should need to know, and often a new language seems to me like “Hey, check this out!  Here’s a more complex or non-intuitive way to accomplish a task you already know how to accomplish, but in a language you will never use professionally!”

I know, I’m disappointing you.

But Go!  Oh my, this seems different.  From the blog post:

Go combines the development speed of working in a dynamic language like Python with the performance and safety of a compiled language like C or C++. Typical…

Wait!  Hold on there — my heart just skipped a beat and it freaked me out.  Can you please repeat that again?

…a dynamic language like Python with the performance and safety of a compiled language like C or C++.

Ooh baby.  Someone help me — I’m shaking with anticipation.  A systems programming language that combines Python and C++?  Can it really be?  It seems too good to be true!  My two favorite programming languages combined in one:  It’s like true programming nirvana!

Last week marked three months that I’ve been working at Microsoft.

As I’ve discussed before, making the decision to leave Mozy for Microsoft was not an easy one.  Let’s face it:  I’m not exactly a spritely youth anymore.  I’ve worked at a lot of different companies — and when I say “different,” I also mean, “different from each other:”  Small companies you’ve never heard of (Spillman Technologies), large companies you’ve surely heard of (IBM), companies whose politics continue to keep them from succeeding (Novell), companies who manage to succeed in spite of the politics (Mozy), and companies that just frankly exist only as dark, ghostly nightmares in the frightening nether regions of my mind (Enterasys Networks).  Yet as different as these places are from each other, one thing mostly remains the same:  the process of creating software is the same everywhere.

So that makes a decision to leave hard.  Since the process of creating software is the same everywhere, it is the intangibles that end up mattering, such as whether you like your boss, whether you get a nice computer or monitor, how comfortable your chair is, etc.  When you consider leaving, you wonder what unidentified intangibles you’ll be giving up and what you’ll be getting, and whether you will feel like this was a good trade a year later.

Leaving Novell for Mozy was like this for me.  I got many, but not all, of the intangibles I expected when I went to Mozy.  I gave up all of the intangibles I expected I’d give up from Novell, like five weeks of paid vacation and a beautiful window office on the 7th floor looking directly north to Mount Timpanogos.  Some things at Mozy ended up being worse than I expected, e.g. the 5% pay cut last spring.  Of course, I do realize that it is not Mozy’s fault that I didn’t get all the intangibles I expected; I set that expectation, not them; I failed to assess the situation accurately.

Nonetheless, as I contemplated leaving Mozy for Microsoft, I thought about this.  “Well, software engineering is the same everywhere.  So since the in-and-out of the job function is mostly the same, I wonder what intangibles I’m gaining and what I’m giving up?”

Well, I failed to assess the situation accurately again.  I made one key error:  Software engineering is NOT the same everywhere.

In particular, it is not the same at Microsoft.  At Microsoft, software engineering is more… uh… yeah:  more.

More better.

Have you ever worked for Microsoft?  If you haven’t, you don’t know anything about us.  I know you think you do.  You don’t.

Never in my career have I ever worked in any organization that took software engineering as seriously as Microsoft does.  I was very surprised to see how seriously we consider things like security and software quality.  I’m aware of the reputation Microsoft has received over the years for bugs and security issues.  Maybe things are different now, or maybe that whole thing was just a function of being the world’s largest, most powerful, and most widely used software company.  At any rate, I can tell you from personal experience that security and quality are very important here — important enough that we will delay shipment if we don’t feel like it meets our standards.  While this may seem obvious, I’ve never seen this commitment to quality permeate throughout an organization like it does here.

It has been incredibly refreshing to see a company take software engineering as seriously as I do.  I love that I’m free to require explanation or justification from my management when I don’t understand something.  I love that I’m supported in insisting on perfection in software design, code, and process to the degree that I can help us deliver it.  I love that people can communicate with me honestly and openly without worrying about my feelings, and that I can do the same with them, because, unlike some places I’ve worked, there is an undercurrent of trust and mutual respect between me and all of my peers wherein we know and believe that, despite having different opinions, we are each talented and capable professionals with the best interests of the company at heart.  I love being surrounded by incredible talent that makes me feel both humbled to be a part of the group and inspired to improve myself every day.  I love working for a company where, instead of feeling like my career has topped out and has nowhere else to go, I feel I have broad, wide-open vistas of learning and advancement just laying before my feet; opportunities sitting before me just waiting for me to seize them.

I had no idea a software company could be that much better than what I’d experienced in the past.  It is really awesome.  It may not be for everybody.  Not all software engineers care enough about delivering quality software that they will do whatever it takes — write unit tests, participate in code reviews, follow rigorous and time-consuming processes, be a small fish in a big pond — in order to do it.  But if you care about delivering quality software, like I do, I must say I highly recommend us.

After only three months I find myself saying something I never thought I’d say:  I love working at Microsoft.  I really do.  Intending absolutely no negative to any other company I’ve worked for (with the exception of Enterasys Networks, I have fond memories of great talent, great people, and great product deliveries at every company), working at Microsoft is unlike anything else I’ve ever experienced.

Merriam-Webster’s dictionary defines discipline as:  “…”

Nah.  Just kidding.

Most software teams I’ve worked on have had some degree of discipline around what code changes go into the codebase.  This tends to take the form of:

• Any change should be associated with a defect or feature that is documented somewhere.
• Commit small, isolated, related changes, all associated with a single defect or feature, that can be easily rolled back.
• Test changes before committing them; avoid breaking the build at all costs.

That last one seems most obvious, but it is so critical that it has to be stated anyway.  Think of it this way:  If you work for me on an already functioning product, I would rather have you do NO WORK AT ALL than to break the build.

Of course, I would not pay you to do no work at all, either.  But breaking the build is taking the product in the opposite direction from where we want it to go.

Some time ago I worked on a software team* where a team member had a problem with this kind of discipline.  Details don’t matter too much, but the net effect was that whenever this person made a commit, the rest of the team would tense up and hold their breath as they integrated the updates.  This person had broken the build enough that his/her commits weren’t trusted by the rest of the team.  And even if the code would compile and run, you still didn’t trust the changes because this person would often sprinkle in little changes hither and yon that were not tied to a documented defect or feature, and often wouldn’t even call out the changes in the commit message.

(As a slight distraction here, I’ll point out how frustrating this is when the team member works in the same office as you, as cited above.  When that person works remotely, it becomes MUCH MUCH WORSE.)

I discussed this with my boss one day who wasn’t sure what to do.  We had a person on the team who was obviously very talented, but who was just not following good practice despite repeated requests to do so.  I offered up the suggestion that I’m restating here, which is basically the following:

Disciplined software engineering should be a prerequisite to having the rights to make critical code contributions.

In other words, no matter how talented a programmer is, they shouldn’t be allowed to contribute code to critical parts of the product if they do not exhibit sufficient discipline to be there.  So what are the “critical parts” and what is the “sufficient discipline”?  The critical parts are the parts that, if they are broken, the software doesn’t perform it’s primary function.  For a product like Firefox, the “Help” dialog is almost certainly not critical, but the HTML parsing engine definitely is.  The sufficient discipline is what I outlined above, but primarily centered around not breaking the build.

So to summarize, what it means is, someone who continually breaks the build should be disallowed to commit changes directly to the critical parts of the product until they earn that right back by adopting the discipline required.  Of course, if you are that free-spirited maverick, there’s a pretty strong incentive now to change your approach.  You may not be paid to be disciplined, but you are paid to deliver software, and if you can’t deliver software because your approach is irresponsible, this should encourage you to get on track.

Some teams choose to approach this by building elaborate systems that enforce proper behavior.  I could build a complex checkin proxy that automatically merges my changes with the latest changes in the repository, builds the product, and then runs the tests (there are tests, right?), and then only makes my commit once all of that passes acceptably.  That’s nice and all, I suppose.  But it seems like a lot of infrastructure, overhead, and process to impose upon the whole team, when to me it seems reasonable to expect that professional software engineers have enough discipline to do this on their own.

So if you aren’t doing this now, how do you go about changing?

Well, you’ve gotta have a suite of automated tests in order to do any of this – a “broken” build is defined as one that does not pass the automated tests.  If you do not have any automated tests, it is time for you to repent, my son.  I do not have time to go into this here, but you must confess your grievous sin and atone by writing an automated test and adding it to your build process.  Just one test is enough to get started – but it MUST be available as a part of the normal build process.  Then every time a build passes the test suite but is, in fact, broken, update your test suite.

Now that you have automated tests, insist that your developers do the following before they commit:

• Update.
• Build.
• Test.

See?  It’s easy.  If the tests pass, they can commit.

Now it is time for you to be disciplined.  Committing code that does not pass the tests is a naughty thing to do.  Do not let your developers be naughty.  Take immediate action.  I think I’d take the following approach:

• First time is a meeting with the boss.  Go over the process.  Make sure they understand the process and what happened.  Personally, it seems like it would be tough to misunderstand, but hey, things happen.  Don’t be presumptuous or condescending.  This meeting alone might be enough to get most people in line.
• If the problem continues, require peer review before checkin.
• If the problem continues, another meeting.  This time, express that all of their current assignments have been reassigned to other members of the team, and they are being reassigned work that is not critical to the functionality of the project.  Let them know that you are concerned about the impact this will have on their performance, because they are expected to contribute at a higher level than what their new assignments will allow.  They have to demonstrate changed behavior and commitment to discipline in order to earn the right to do the important work again.
• If the problem continues after that, it is probably performance-improvement-plan time.  Or mutual-agreement-that-they-find-a-different-employer time.

*(To my faithful readers who think they know what situation I’m speaking of, may I remind you that I’ve worked for seven different software companies and thirteen-ish different software teams over fourteen-plus years in this career, so be careful what assumptions you might make.  Thank you.)

It took me a while, but I finally settled into my routine and got to where I’m reading my RSS feeds most days again.  I was going through the posts of the past month or so, since the job change, and ran across this article on the “Making Good Software” blog about things that keep someone from being a good software engineer, outside of (and often in spite of) an ability to engineer software.

I’ll summarize here.  It isn’t my intent to plagiarize; if you are remotely interested go read the article.  Here are the things:

• Lack of discipline
• Big ego
• Poor communication
• Forgetting the customer
• Lack of proper work prioritization

I have known many of these people during my career.  Indeed, I was one of them.  I remember coming to Novell from IBM almost ten years ago.  I thought I was pretty hot stuff and I made sure my team knew it.  In fact, I actually said (this is embarrassing to admit) on more than one occasion, “There are people who know C++ better than I do, but I haven’t met any of them.”  My ego surely made me hard to work with.  It definitely was a cause of friction between myself and my management chain, and ended up being a (deserved) source of frustration and difficulty for me, until I recognized the problem and started working to address it.

I’m pretty ashamed of having behaved that way back then.  I hope I’m better than that today.  I guess recognizing the weakness is a good first step.  Fortunately for me, back then I was on a really great team with a lot of very capable, patient, and talented engineers that waited for me to learn from my mistakes and to grant them the mutual respect they deserved.  I consider myself pretty fortunate to have been able to learn from them what real software engineering is about.

Over my career I’ve had to work with people like this from time to time, software engineers that manifest one or more of these traits.  Sometimes these guys are pretty talented technically.  I’ve felt sorry for them as I’ve observed, realizing that these weaknesses are going to hold their career back until they recognize them and work to overcome them.  No amount of programming prowess will compensate for it.  And what’s even worse is, often because these people have the personality issues they have, you don’t get anywhere by trying to bring these weaknesses to their attention; they are often unreceptive to this type of feedback.  Like I said, you just have to wait until they recognize it themselves.

I can imagine being in a performance review with someone like this, having them explain to me all the technical awesome they did, and me replying, “Your poor soft skills are shouting so loudly that I cannot hear your technical awesomeness.”  Or, as I said in the title, actions speak louder than code.

I really believe this is true.  To write software professionally, of course you must have technical ability; however, this is a necessary but not sufficient condition for greatness.  The best software engineers I’ve had the fortune to work with in my career, past and present, not only had awesome technical ability but did not exhibit weakness in these areas.  And I’ll tell you what:  Those teams are wonderful teams to be a part of.  Those teams create strong. uplifting work environments and are able to deliver great products that meet customer demand.

Another way to say this is, in order to be a good software engineer, you must first be a good employee.

In fact, I’ll tell you how important I think this is.  The ability to mitigate or eliminate these defects from a software engineer’s persona is so important to me that, if I had my own company and were making the hiring decisions, I would not hire a candidate that I knew had these problems, no matter how incredible their technical ability.

A person with these weaknesses is really only suited to be set to the side to work on a special side R&D project where interaction with other employees is limited, and they don’t have to interact with customers at all.  Problem is, those kind of projects are either a) strategically important to the long-term future of the company, or b) of little to no real value, or c) a combination, often high potential value but with a lot of inherent risk that causes the real value to be low.  If the project is strategically important or of high value, do you really want to reward the biggest jerk in your company by giving him the highest profile assignment, leaving your best engineers to maintain the legacy project?  Wouldn’t you want to have someone working on that high profile assignment that knows how to collaborate with others and assemble all the best ideas to solve the problem the best way, even if that solution isn’t his/her own?  Contrariwise, if the project is of little real value or has so much risk that it offsets the real value, why even do it at all?

Nope.  In my company, if I were ever to have one, I wouldn’t hire or keep an employee who had these weaknesses and was not committed to addressing them.  I’ve seen the difference, both in morale and productivity, between teams where they don’t have these problems and teams that do.

The software development community is abuzz because of this article written by Tom DeMarco, author of that seminal work “Peopleware” that I ordered from some joker on eBay about six weeks ago but has not come yet.

You really should read the article.  I say this because I assume that you are a software engineer, like me.  Otherwise, let me summarize it for you.

Nah, that’ll take too long.  Let me sum up.

DeMarco basically says in the article the best software development on the most valuable software products with the highest degree of utility happens in the absence of control and measurement.  In other words, everything every software engineer on earth was taught about software engineering in college is wrong.

Yeah, that’s pretty much what he’s saying.

This has led to a lot of discussion around what we call all those people that we used to call software engineers just last month.  The impetus is that if the best software is created by not managing or quantifying the work and the deliverables in the project, is this really engineering per se?

Well, I’ve known this for a long time.  I just haven’t said anything about it.  Seriously, stop laughing.  I don’t expect you to believe me.  In fact, I’m happy to give credit to all the other people.  But it is certainly true that, although I’ve had that word in my job title for well over a decade now, I’ve felt for some time that software engineers are unique from other types of engineers.  Indeed, good “software engineering” requires creativity and artistry as much as it requires engineering.  “Software craftsmanship” is the term Jeff Atwood at Coding Horror is pushing today; “Software Artisan” is another term I’ve heard today as the new replacement for “Software Engineer.”

Whatever.  I know what it is I do, so you can call me a Software Masseur if you feel like you must.  I’ll just keep doing my job like always.

Joking aside, this article is a big deal because someone (more specifically, someone people will listen to) has finally come out and said what we’ve felt all along – trying to measure and control the creation of software is not helping, primarily because it is at least as much a creative process as it is a technical one.  Convincing the Employees Formerly Known As Software Engineers around the world this is true is not going to be a hard sell.

Convincing their employers and customers, on the other hand, is going to be tough.  That’s because the customers have come to expect that they will know what features will be in what products by what date, and they aren’t likely to give that up willingly.  They need (or think they need) to know all of this so they can plan – when to buy their next computer, when to get trained on the new version of a product, when to rollout system updates to an enterprise, when a partner can promise the delivery of new functionality to their customers, etc.

Interestingly, some organizations are able to avoid this expectation of their customers.  Open source projects are particularly good at this.  Eclipse, for example, delivers a release every year but makes no promise as to what features will be included in that release; other projects, like Linux and Apache, release whenever they feel like it.

I remember writing a software product that was going to be delivered as Apache modules for Apache 2.0, way back before Apache 2.0 was officially released.  We e-mailed the Apache project during development to ask when Apache 2.0 was scheduled to ship, and they said in essence, “We don’t know.  And, we don’t publicize or even set release date goals.  We ship it when it is ready.”  Of course that made it hard to plan, but part of me secretly wished I could get paid to work on Apache.

Open source projects are not the only ones that do this, however.  I defy you to figure out when any given Google app will ship, for example.  And consider Blizzard Software, one of the most successful video game companies EVER (ever heard of World of Warcraft?).  As I understand it, they are like Apache in this regard – their product will ship when it is ready, and not a day before, and it doesn’t matter what big holiday or trade show is coming up.  And so far they’ve done alright.

And then there’s Duke Nukem Forever.  Wait…

So Amber and I are waiting at Salt Lake International for our flights to Seattle, and I see this guy walk in front of me. I leaned over to Amber and said, “That guy right there looks a lot like an old college professor I had. The one right there with the curly hair. I wonder if it is him.”

I didn’t say anything though, until later on the plane, where as luck would have it his seat was right next to mine. After we both sat down, I said, “You look like someone I know.”

“I doubt it,” he said.

That’s a weird thing to say, because how could he possibly who all of the people that I know are, and what they look like? And whether I think one of them looks like him? But whatever.

“Yeah, well, I had a college professor that looked a lot like you.”

“Oh, really? What school?”

“Utah State. Do you teach there?”

“Um, yeah.”

“Are you —- —- ?”

“Yes.”

I’ve removed his name here to protect his identity, because you cannot look on USU’s website and go through the faculty that teach in the Computer Science department and figure out which one has curly hair and has a name that rhymes with a scheme for doing something revolting*, or what you might do to another person to annoy them**, or the description of someone who is really, really passionate about masonry***, or a law that prohibits people from using a naked finger to remove debris from their nose****, or that really sweet ride that Frankie (aka Summer George) got for Jerry Seinfeld*****.

So anyway, I pointed out that I know him because I took a class from him.  He asked just enough to see if I’d made anything with my life, which pretty much means, did I become a software engineer in the field of artificial intelligence, which is his passion.  And since I just became the kind of software engineer that actually makes money, he quickly lost interest.

At this point his face clearly said, “Go away.”  So I did.  I mean, I had to sit next to him for the rest of the flight.  But I tried not to touch him.

Sheesh dude, it isn’t like I was trying to get your autograph or anything.

*sick plan
**flick man
***brick fan
****pick ban
*****trick van

Setting up a new computer is one of those things that should be enjoyable, but is mostly just tedium.  That’s because there really isn’t a single OS out there that does for me everything I want in a single distribution – at least not one I’ve found.  In truth Linux comes closest, but in the case of Linux, there are still some things (like Motocross Madness 2, one of the best PC games ever) that you just don’t get there.

And don’t start giving me lectures on Mac.  Same problem applies there.  Even without games, I still have pretty much the same setup overhead for Mac as anything else.  Macs are great, don’t get me wrong, but I don’t write Mac software anymore, so I don’t have to be showing the Steve-love for a while now, until I start doing Mac development again.

Anyway, I’m willing to bet that at my new job my development machine will be a Windows machine – it’s just a hunch I have.  So here’s the rundown of setting the machine up for usefulness and effectiveness.

Basics
	First is Firefox. Firefox is a great web browser, fast and pretty reliable.  Once you’ve got Firefox installed, you’ll want to grab a handful of Firefox plugins.  When I set up next, I’ll be trying Google Gears, AdBlock, FireBug, Better GMail/GCal/GReader, Tab Mix Plus, FaviconizeTab, Fission, and GreaseMonkey of course.  I keep IE around because sometimes I need it, but I make Firefox my default browser.
	In addition to e-mail, I use calendaring and news readers (RSS/Atom) almost every day. A lack of decent free options in the past got me used to using Google for all of this stuff. But if I were to decide to use rich applications for these purposes instead, I’d give Thunderbird, Sunbird, and RSSOwl a try.
	Next is OpenOffice.org. I know, most people use Microsoft Office.  I realize it is better.  I realize it is more powerful.  I realize it is more ubiquitous.  It is also expensive for my purposes.  Stick with Office if you like it.
	For instant messaging I use Pidgin. Since I’ve got friends using MSN/Hotmail, Google, and Yahoo! among others, Pidgin gives me a great way to be able to chat with all of them in a single IM client.  And it has some great plugins that I’m eager to try out.
Multimedia
	For listening to and organizing music, I’ve been hearing a lot about Songbird and I think I’ll give that a try.  I don’t buy music from iTunes and I don’t have an iPod (I know, lame).  If I did I’d go with iTunes.  Although, I do like iTunes Genius feature, so I might go with iTunes just for that.
	When it comes to audio editing, Audacity is where it’s at.  I’ve used Audacity to make ringtones from some of my music MP3s, to edit and mix recorded WAV files into MP3 files, and even for my son’s science project to examine the differences between sound waves.  A must-have.
	In doing research for this blog post, I ran across these apps:  MediaCoder for translating and saving media files, Handbrake for ripping copies of your DVDs to formats for your handheld, and ImgBurn for creating DVDs.  So I haven’t actually used them yet, but I can hardly wait to try them out.  Managing video files and recordings is something I built my computer to do, but finding the software to get the job done has been tough.  Hopefully I’ll find the answer among these tools.
	DoubleTwist is a new application I’m eager to try for managing the transfer and synchronization of files from the PC to your handheld device.  I’ve got a really cool little Sony Ericsson phone that is supposed to work flawlessly with DoubleTwist; can’t wait to find out.
	I’ve used The Gimp for my photo editing for years and, for me, it removes any need for me to buy Photoshop.  I’m sure Photoshop users would disagree.  But hey, I’m not a graphic artist.  I’m just a guy who needs to edit photos from time to time, even for my job, and can’t justify the expense of Photoshop.  Lately, friends have been telling me about Paint.NET, and what they are telling me is that they like it better than The Gimp.  That’s a high standard in my opinion.  I’ll have to check it out.
	Ah – where would I be without Steam?  Steam’s client is free to download and serves as the launching pad for most of the games I play.  A lot of games I really like, such as Audiosurf and World of Goo, I first found out about via Steam.  Every time I launch it it seems there are more titles and more publishers available through Steam.
Security
	Truecrypt is a highly regarded application for encrypting data on your PC – one I’ve been meaning to try for a long time and plan to soon.  I know, I should do this.  Eraser, on the other hand, is one I’ve used for a long time.  It makes it really easy to truly erase files from your computer by performing multiple overwrite passes to keep your data from being restored after you’ve intentionally deleted it.
	For managing the applications that launch automatically when your PC starts, it is hard to beat Mike Lin’s Startup Monitor and Startup Control Panel applications.  Startup Monitor runs discretely in the background, and just notifies you when some application has requested to be run at startup, allowing you to decide whether to accept this or not.  Startup Control Panel offers a simple view of the applications already scheduled to run at startup, and allows you to disable them.  Great for improving boot times and free resources, not to mention helpful in keeping your desktop secure from rogue apps running in the background doing who knows what.
	Spybot Search & Destroy is essential for keeping your PC clear of spyware and adware that want to do evil things behind your back.  It integrates with most common web browsers, including Firefox, to help lock them down to avoid evil cookies and other tracking software from sending information about you to others.
	If you are up for trying a free PC antivirus application, ClamAV is the answer for you. Okay, I haven’t used it and probably won’t for a while because I’ve already got a license for a security suite.  But if you are in the market it is probably worth a try.  It certainly could not be worse than BitDefender (that steaming pile).
Utilities
	Daemon Tools is a simple utility that can mount local disk images as filesystems.  Mac does this very easily, of course, with .dmg files, but you need a tool like Daemon Tools to do it on a PC.  Use Alcohol 120% to create mountable disk images from game CDs, for example, which will enable you in most cases to play PC games by mounting the disk image in Daemon Tools instead of inserting the CD.  Or use it to mount ripped DVD ISOs when you are converting them to a format you can use on your handheld.
	Many years ago, Novell had this really great product called iFolder that you used to synchronize files between multiple computers.  Like many Novell products, it was a really awesome product that nobody ever heard about because Novell can’t figure out how to market anything.  But Novell employees know about iFolder and most of them are like me – once I got used to using it I could hardly stand to not have it. When I left Novell this was a big big problem for me. Finally Dropbox came along to address my problem.  Dropbox allows you to do what iFolder did years ago – synchronize files between multiple computers.  Dropbox is not nearly as full featured as the latest iFolder 3, but at least this one you can use without being a Novell employee.
	As I’ve said before, once I started working for Mozy I realized that online backup should be considered essential for anyone.  I really don’t know why a person wouldn’t use Mozy.  Even if you are backing your data up on a second drive, USB drive, thumb drive, etc. you should be using Mozy, to automatically provide a secure backup copy of your data in a separate location – for recovery from fire damage, for example. Having worked at Mozy for the past 14 months, I can vouch for their solid technology which is, in my opinion, the best in the industry without question.  You need online backup, so why not use Mozy?  You can back up 2GB for free or as much as you want for $5/month.
	For archiving and compressing infrequently-used data, I recommend 7Zip. It will unpackage almost anything and will package in the most common formats, including Linux-compatible TAR/GZ formats.  It’ll also do encryption and self-extracting packages in some formats.  In other words, it’s pretty much everything you want in an archiving tool, for no cost.
	I haven’t tried Everything yet but I plan to soon.  This is a highly rated desktop search engine along the lines of Spotlight for Mac.  Windows search I mostly use as last resort, but if this is anything like Spotlight I’ll use Everything all the time.
	Freemind is a note-taking application that I’m eager to try.  I’ve been needing one of these for some time, so I’m anxious to give it a shot.
	I consider Cygwin an essential PC utility.  Since I’m a lot more familiar with the Linux shell than the PC DOS-style shell, Cygwin provides me with a command prompt I’m comfortable with.  Cygwin comes with a large number of helpful tools, like the GNU C complier suite, ssh/scp, wget, and others.
	I’d also consider SQLite an essential PC utility.  SQLite is a very simple file-based SQL engine that is very useful and freaking awesome.  I’d recommend a PC utility for using SQLite but there really isn’t a good one.  Probably your best option is SQLiteSpy.
Application Development
	I’ve said before that if a person’s going to learn to write software, I think the two most important languages to learn are C and Python.  Since you already installed Cygwin you probably already have a C compiler on your machine, so now you need to get Python installed. In addition to Python, Ruby seems pretty interesting and one you should definitely look at, in addition to Python (and not instead of Python, not yet anyway). You should note that if you plan to do Python and/or Ruby development, you’ll probably want to get used to doing that natively on your machine, and not via Cygwin.  So don’t depend on the Cygwin Python and Ruby interpreters – use the native interpreters instead. If you must, use Java.  Sometimes there’s stuff you just can’t do without it.
	Ah, Eclipse:  the mother of all development environments.  Having worked closely with the Eclipse foundation and been part of starting an Eclipse project myself, I have a strong affinity for Eclipse.  For Java development, I’d consider it one of the best, if not the best, Java IDE available.  It’s also a great free alternative for a lot of other languages and application types.  Get not only the base Eclipse, but the plugins for C/C++, PHP, RCP/Plugin development, Data Tools, Test and Profiling Tools, and Web Tools. And if you aren’t going to get the Eclipse Python plugin, you’ll want to install Eric instead.  Eric is a pretty good little Python IDE that works on both Linux and Windows.  You’ll need PyQt for Eric to work I think. If you are really wanting to do development in C# and .NET instead, but don’t have the .NET platform, you could try SharpDevelop.  I haven’t tried it though, so I can’t say – and in my new job, I’ll be doing my C# development in Visual Studio, which is certainly better.
	If you want to try out simple GUI programming, especially cross-platform GUI programming, try wxWidgets.  You can program directly to wxWidgets in C or C++, or in Python using wxPython.  If you think GUI programming with wxPython is your cup of tea, you might also want to try Boa Constructor, which is a good Python development environment with GUI building tools.
	For web development on a Windows PC, I love WAMP.  This simple bundle offers Apache, MySQL, and PHP all together in a single package that you can easily start and stop all as one. (This is the part where the Mozy PHP bigots comment to tell me how rotten PHP is, and where they tell me how much better Perl is, and where I nod and pretend to agree in order to keep the peace.  So bring it on.) WAMP is great for your typical free-style web application development, especially if you are building from an existing framework, which is quite likely to be built in PHP.  If you’re building from scratch, however, you would probably want to strongly consider Ruby on Rails, in which case you’ll want to install RubyGems to get Rails and other goodies. Finally, I hear Kompozer is a pretty good HTML-style editor and page builder, and I might give that a shot sometime.

Uh, this is a programming post.  Just so you know.

The Problem

Okay, so let’s start this discussion with a simple linked list that contains the ingredients for Bill Cosby’s Chocolate Cake for Breakfast:

Fig. 1 - Linked list of ingredients for Chocolate Cake for Breakfast

Normally we’d use a linked list in programming when we know we may have to arbitrarily insert items later on.  For example, I just realized that this cake does not contain chocolate, so I should be able to insert that into the list by shuffling a little bit:

Fig. 2 - Chocolate Cake for Breakfast, with Chocolate Added

This seems like a much tastier recipe.  But there can be a slight problem with this.

Suppose the linked list is serving as a model for your recipe view.  You have a UI that accessess the model and then displays the results in some sort of view that you can see on your laptop in the kitchen.  Suppose further that you decide to go even crazier and not just stop with adding chocolate, but you are also going to add sugar and frosting.  INSANE!

This is not a problem yet.  Likely your UI offers the ability to add new ingredients through the UI, so you just add them and the model gets updated, and when your view refreshes you see the new list:

Fig. 3 - Chocolate Cake for Breakfast - Dessert Style!

That sounds like a great chocolate cake.  But now suppose Mr. Cosby is on stage in Winnemucca doing his Chocolate Cake for Breakfast routine, and for some reason that nobody can understand the teleprompter that reminds him of the routine is using your data model to render the view – right at the exact moment in time that you are changing the recipe.

Uh-oh.

I’ve run into this problem a number of times in my software development career, where I have a data structure that is being accessed by more than one user at the same time.  In our case, we have one read-only accessor and one read/write accessor.  At first glance it doesn’t seem like this should be a problem.  But unfortunately, even though the reader isn’t making changes to the model, the reader still depends on the model having a constant state.

There’s a couple of fairly simple ways to address this problem.

First, we can make a copy of the model for each user.  We can have a resource that provides a copy when a user requests the copy.  Then that user can do whatever they want with the model – read or write.  When they are done, they can either discard their copy, or turn it back to the resource for the changes to be merged into the whole.

This might work for some implementations.  In our trivial example here it would probably be just fine.  But if I have a data structure with two million 50-byte records, this isn’t such a great idea.  It isn’t just that I don’t want to make a copy of a 100 Mbyte (okay, 95 for you technicality folk) data structure – I don’t want to double my memory usage from 100 Mbytes to 200 Mbytes (or 95 to 190, sheesh).  So making a copy won’t always work.

Another option is to lock the structure.  Using some sort of a mutual exclusion primitive I can lock the data structure so that only one user can access it at a time.  I can’t just hold the lock per-access, though – I have to maintain sessions and locks for the entire session.  In other words, I have to make sure that nobody can change the recipe at all throughout the duration of Mr. Cosby’s act.  This high level of granularity in the locking of the data structure makes it hard to use.  And it is particularly frustrating for read-only users who are thinking, “Why can’t I access the structure?  All I want to know is what is in it!”

Versioning

One way I’ve successfully solved this problem in production code is to use a versioned container and versioned iterators for that container.  I used these at Volera, for example, for some reason that doesn’t matter now, because Volera is DEAD.  Anyway, here’s how they work:

Versioned Containers act pretty much like regular containers – vectors, lists, hashtables, etc.  The difference is that each link from one element in the container to the next has an associated version number on the link.  Any given element can have a number of “next” links – even a singly linked list, like in our example.  Each link would have a different version number.  The container itself also has a version number, which corresponds to the highest numbered link in the container.

Versioned Iterators go along with the versioned container.  When you request an iterator from a container, it gives the iterator a version number which corresponds to the current version number of the container at the time of request.  The user requesting the iterator can also request a different version number.

Here’s what our data structure might look like if it were versioned:

Fig. 4 - Chocolate Cake for Breakfast with Versioned Containers and Iterators

This ends up working out pretty cool.  Mr. Cosby can request a versioned iterator of the view when his show begins that displays the model as it is when his show starts.  Let’s say that his iterator is version 1.  That iterator traverses the list by only following links where the version number is less than or equal to the version number on the iterator.  So it first goes to “eggs”, then follows link 1 to “milk”, then follows link 1 to “wheat”, then ends.  It can be doing this at the same time that another user is changing the data structure, first by inserting “chocolate,” which creates the version 2 link from “milk” to “chocolate” and the version 2 link from “chocolate” to “wheat,” and later by inserting “sugar” and “frosting” as version 3.  A version two iterator on the structure would start with “eggs” then follow the version 1 link (remember – less than or equal to the iterator version) to “milk.”  But then it would follow the version 2 link instead of the version 1 link, so now it goes to “chocolate” instead of going directly to “wheat.”

The same goes for the version 3 iterator, which gives us everything in the data structure.

There’s a couple of issues you have to deal with when you implement a structure like this.  First off, you need some sort of a scheme to go through the structure and prune it of old traversal versions.  If I remember correctly, a scheme that will work to do this is to have the container itself remember how many users are accessing it and what versions they are at.  When one iterator goes away, the iterator should report this back to the container.  The container then looks to see if that iterator is the lowest-version-number iterator that it knew about.  If it is, the container can go through and prune all of the links that are versioned at that number and below, as well as nodes in the structure that are only included in the structure by links at that version and below.  So there’s a little bit of housekeeping that has to go on, but the container should be able to do this after iterators go away.

In other words, once Mr. Cosby’s show ends, the data structure that previously looked like Fig. 4 should end up looking more like Fig. 3.

Another problem to consider is this:  If the data structure lives for a long time and has lots of accessors, it is possible for the version numbers to overflow.  The overflowing isn’t necessarily a problem, though, as long as you keep track of what is really the oldest version in the data structure.  Another way to do this is, as you prune the table, you can take the liberty of reversioning the table if there are no accessors to the table.  Depending on your situation, you may never have a chance to lock the whole structure in order to reversion it, but the first option should be workable.

Lastly, since you have to version the links themselves, you might not be able to directly use standard containers; for example, if you are using C++ you might have to create your own linked list template with a list of versioned next pointers in each node, instead of implementing your versioned list container in terms of the STL list container.  I never tried this so I’m not sure how it would work.  When I did this, I did it using a tree container that was implemented in STL style, with versioning in the pointers and iterators.  It was pretty cool.