The Go Gopher

Earlier this week Google announced their new programming language, called Go.

Usually I don’t get too worked up about programming languages.  I already feel like I know more languages than I should need to know, and often a new language seems to me like “Hey, check this out!  Here’s a more complex or non-intuitive way to accomplish a task you already know how to accomplish, but in a language you will never use professionally!”

I know, I’m disappointing you.

But Go!  Oh my, this seems different.  From the blog post:

Go combines the development speed of working in a dynamic language like Python with the performance and safety of a compiled language like C or C++. Typical…

Wait!  Hold on there — my heart just skipped a beat and it freaked me out.  Can you please repeat that again?

…a dynamic language like Python with the performance and safety of a compiled language like C or C++.

Ooh baby.  Someone help me — I’m shaking with anticipation.  A systems programming language that combines Python and C++?  Can it really be?  It seems too good to be true!  My two favorite programming languages combined in one:  It’s like true programming nirvana!

I’ll summarize here.  It isn’t my intent to plagiarize; if you are remotely interested go read the article.  Here are the things:

• Lack of discipline
• Big ego
• Poor communication
• Forgetting the customer
• Lack of proper work prioritization

I have known many of these people during my career.  Indeed, I was one of them.  I remember coming to Novell from IBM almost ten years ago.  I thought I was pretty hot stuff and I made sure my team knew it.  In fact, I actually said (this is embarrassing to admit) on more than one occasion, “There are people who know C++ better than I do, but I haven’t met any of them.”  My ego surely made me hard to work with.  It definitely was a cause of friction between myself and my management chain, and ended up being a (deserved) source of frustration and difficulty for me, until I recognized the problem and started working to address it.

I’m pretty ashamed of having behaved that way back then.  I hope I’m better than that today.  I guess recognizing the weakness is a good first step.  Fortunately for me, back then I was on a really great team with a lot of very capable, patient, and talented engineers that waited for me to learn from my mistakes and to grant them the mutual respect they deserved.  I consider myself pretty fortunate to have been able to learn from them what real software engineering is about.

Over my career I’ve had to work with people like this from time to time, software engineers that manifest one or more of these traits.  Sometimes these guys are pretty talented technically.  I’ve felt sorry for them as I’ve observed, realizing that these weaknesses are going to hold their career back until they recognize them and work to overcome them.  No amount of programming prowess will compensate for it.  And what’s even worse is, often because these people have the personality issues they have, you don’t get anywhere by trying to bring these weaknesses to their attention; they are often unreceptive to this type of feedback.  Like I said, you just have to wait until they recognize it themselves.

I can imagine being in a performance review with someone like this, having them explain to me all the technical awesome they did, and me replying, “Your poor soft skills are shouting so loudly that I cannot hear your technical awesomeness.”  Or, as I said in the title, actions speak louder than code.

I really believe this is true.  To write software professionally, of course you must have technical ability; however, this is a necessary but not sufficient condition for greatness.  The best software engineers I’ve had the fortune to work with in my career, past and present, not only had awesome technical ability but did not exhibit weakness in these areas.  And I’ll tell you what:  Those teams are wonderful teams to be a part of.  Those teams create strong. uplifting work environments and are able to deliver great products that meet customer demand.

Another way to say this is, in order to be a good software engineer, you must first be a good employee.

In fact, I’ll tell you how important I think this is.  The ability to mitigate or eliminate these defects from a software engineer’s persona is so important to me that, if I had my own company and were making the hiring decisions, I would not hire a candidate that I knew had these problems, no matter how incredible their technical ability.

A person with these weaknesses is really only suited to be set to the side to work on a special side R&D project where interaction with other employees is limited, and they don’t have to interact with customers at all.  Problem is, those kind of projects are either a) strategically important to the long-term future of the company, or b) of little to no real value, or c) a combination, often high potential value but with a lot of inherent risk that causes the real value to be low.  If the project is strategically important or of high value, do you really want to reward the biggest jerk in your company by giving him the highest profile assignment, leaving your best engineers to maintain the legacy project?  Wouldn’t you want to have someone working on that high profile assignment that knows how to collaborate with others and assemble all the best ideas to solve the problem the best way, even if that solution isn’t his/her own?  Contrariwise, if the project is of little real value or has so much risk that it offsets the real value, why even do it at all?

Nope.  In my company, if I were ever to have one, I wouldn’t hire or keep an employee who had these weaknesses and was not committed to addressing them.  I’ve seen the difference, both in morale and productivity, between teams where they don’t have these problems and teams that do.

The Problem

Okay, so let’s start this discussion with a simple linked list that contains the ingredients for Bill Cosby’s Chocolate Cake for Breakfast:

Fig. 1 - Linked list of ingredients for Chocolate Cake for Breakfast

Normally we’d use a linked list in programming when we know we may have to arbitrarily insert items later on.  For example, I just realized that this cake does not contain chocolate, so I should be able to insert that into the list by shuffling a little bit:

Fig. 2 - Chocolate Cake for Breakfast, with Chocolate Added

This seems like a much tastier recipe.  But there can be a slight problem with this.

Suppose the linked list is serving as a model for your recipe view.  You have a UI that accessess the model and then displays the results in some sort of view that you can see on your laptop in the kitchen.  Suppose further that you decide to go even crazier and not just stop with adding chocolate, but you are also going to add sugar and frosting.  INSANE!

This is not a problem yet.  Likely your UI offers the ability to add new ingredients through the UI, so you just add them and the model gets updated, and when your view refreshes you see the new list:

Fig. 3 - Chocolate Cake for Breakfast - Dessert Style!

That sounds like a great chocolate cake.  But now suppose Mr. Cosby is on stage in Winnemucca doing his Chocolate Cake for Breakfast routine, and for some reason that nobody can understand the teleprompter that reminds him of the routine is using your data model to render the view – right at the exact moment in time that you are changing the recipe.

Uh-oh.

I’ve run into this problem a number of times in my software development career, where I have a data structure that is being accessed by more than one user at the same time.  In our case, we have one read-only accessor and one read/write accessor.  At first glance it doesn’t seem like this should be a problem.  But unfortunately, even though the reader isn’t making changes to the model, the reader still depends on the model having a constant state.

There’s a couple of fairly simple ways to address this problem.

First, we can make a copy of the model for each user.  We can have a resource that provides a copy when a user requests the copy.  Then that user can do whatever they want with the model – read or write.  When they are done, they can either discard their copy, or turn it back to the resource for the changes to be merged into the whole.

This might work for some implementations.  In our trivial example here it would probably be just fine.  But if I have a data structure with two million 50-byte records, this isn’t such a great idea.  It isn’t just that I don’t want to make a copy of a 100 Mbyte (okay, 95 for you technicality folk) data structure – I don’t want to double my memory usage from 100 Mbytes to 200 Mbytes (or 95 to 190, sheesh).  So making a copy won’t always work.

Another option is to lock the structure.  Using some sort of a mutual exclusion primitive I can lock the data structure so that only one user can access it at a time.  I can’t just hold the lock per-access, though – I have to maintain sessions and locks for the entire session.  In other words, I have to make sure that nobody can change the recipe at all throughout the duration of Mr. Cosby’s act.  This high level of granularity in the locking of the data structure makes it hard to use.  And it is particularly frustrating for read-only users who are thinking, “Why can’t I access the structure?  All I want to know is what is in it!”

Versioning

One way I’ve successfully solved this problem in production code is to use a versioned container and versioned iterators for that container.  I used these at Volera, for example, for some reason that doesn’t matter now, because Volera is DEAD.  Anyway, here’s how they work:

Versioned Containers act pretty much like regular containers – vectors, lists, hashtables, etc.  The difference is that each link from one element in the container to the next has an associated version number on the link.  Any given element can have a number of “next” links – even a singly linked list, like in our example.  Each link would have a different version number.  The container itself also has a version number, which corresponds to the highest numbered link in the container.

Versioned Iterators go along with the versioned container.  When you request an iterator from a container, it gives the iterator a version number which corresponds to the current version number of the container at the time of request.  The user requesting the iterator can also request a different version number.

Here’s what our data structure might look like if it were versioned:

Fig. 4 - Chocolate Cake for Breakfast with Versioned Containers and Iterators

This ends up working out pretty cool.  Mr. Cosby can request a versioned iterator of the view when his show begins that displays the model as it is when his show starts.  Let’s say that his iterator is version 1.  That iterator traverses the list by only following links where the version number is less than or equal to the version number on the iterator.  So it first goes to “eggs”, then follows link 1 to “milk”, then follows link 1 to “wheat”, then ends.  It can be doing this at the same time that another user is changing the data structure, first by inserting “chocolate,” which creates the version 2 link from “milk” to “chocolate” and the version 2 link from “chocolate” to “wheat,” and later by inserting “sugar” and “frosting” as version 3.  A version two iterator on the structure would start with “eggs” then follow the version 1 link (remember – less than or equal to the iterator version) to “milk.”  But then it would follow the version 2 link instead of the version 1 link, so now it goes to “chocolate” instead of going directly to “wheat.”

The same goes for the version 3 iterator, which gives us everything in the data structure.

There’s a couple of issues you have to deal with when you implement a structure like this.  First off, you need some sort of a scheme to go through the structure and prune it of old traversal versions.  If I remember correctly, a scheme that will work to do this is to have the container itself remember how many users are accessing it and what versions they are at.  When one iterator goes away, the iterator should report this back to the container.  The container then looks to see if that iterator is the lowest-version-number iterator that it knew about.  If it is, the container can go through and prune all of the links that are versioned at that number and below, as well as nodes in the structure that are only included in the structure by links at that version and below.  So there’s a little bit of housekeeping that has to go on, but the container should be able to do this after iterators go away.

In other words, once Mr. Cosby’s show ends, the data structure that previously looked like Fig. 4 should end up looking more like Fig. 3.

Another problem to consider is this:  If the data structure lives for a long time and has lots of accessors, it is possible for the version numbers to overflow.  The overflowing isn’t necessarily a problem, though, as long as you keep track of what is really the oldest version in the data structure.  Another way to do this is, as you prune the table, you can take the liberty of reversioning the table if there are no accessors to the table.  Depending on your situation, you may never have a chance to lock the whole structure in order to reversion it, but the first option should be workable.

Lastly, since you have to version the links themselves, you might not be able to directly use standard containers; for example, if you are using C++ you might have to create your own linked list template with a list of versioned next pointers in each node, instead of implementing your versioned list container in terms of the STL list container.  I never tried this so I’m not sure how it would work.  When I did this, I did it using a tree container that was implemented in STL style, with versioning in the pointers and iterators.  It was pretty cool.

Honestly, that is only sort of true.  Some of the emails say, “Lose 30 pounds in 12 minutes!”  And some say, “I am the prince of Nigeria!”  And some say, “I am lonely and looking for a purely physical relationship!”

More honestly, that pretty much comprises the bulk of the email I receive.  Nobody has actually asked how to be more like me, strangely enough.  But that day is surely coming, and since I am a computer scientist, or at least since USU says so, I could start with a list of books that you can read if you want to become a true nerd and rule the world the way I do, which is to say, not.

(So, in fairness, I haven’t actually read all of these or owned them. There are some that I haven’t read, but I read one like it; those are marked in blue. There are some that I haven’t read but think I should; those are marked in gray. There are some that where I read one like it, but I want to read that particular one – they are bluish-gray. And I didn’t list the probably 40-50 CS books I own or have owned and read that are not shown here. So cut me a break.)

Math and English

First off, in order to be a good software engineer and computer scientist, you have to be a good mathematician and a good writer.  Sorry.  You simply can’t be a competent software engineer without a solid mathematical background, and you can’t be an effective one if you can’t figure out how to express your ideas clearly in writing.

Learning to Program

Your next step is to learn basic programming concepts.  In my opinion, you should learn two languages at this point:  Python and C.  Python is a good beginning language, very easy to create real applications, easy to learn, and very versatile and useful in the real world.  C is the fundamental systems programming language.  Knowing Python and C will allow you to program just about anything and gives you a good fundamental background.

Computer Science Fundamentals

Having learned how to write basic computer programs, now is time to get into the science of computer science.

Programming Technique and Methodology

How to write software well.

General Programming

Two other languages you might want to know are C++ and Java.  C++ is much maligned, but widely used, especially for systems applications, games, and other high-performing software applications.  Java is an abomination in every sense of the word.  But it is also very popular and good to know.  If you are going to learn Java, you should also learn JNI, so you can get from Java back to C and get some real work done.

Systems Programming

If you are going to do systems programming, you’ve got to know the specifics of how to program to the environment in question.  It’s worth noting here that the UNIX books basically cover POSIX, which applies not only to UNIX but BSD, Linux, and Mac as well to varying reasonable degrees.  I’ve also included an internals book for the big three platforms (Windows, Linux, and Mac).  And if you are going to program for Mac, you will probably want to learn another language: Objective-C.

UNIX/Linux
Windows
Mac OS X

Other

Every good software engineer should clearly understand open source; hence The Cathedral and the Bazaar. You will find you are missing out on a number of inside jokes if you don’t read The Hitchhiker’s Guide to the Galaxy. And everyone should read The Code Book, simply because it is so interesting.

Image Credits: amazon.com and barnesandnoble.com

Anyway, we’re hiring.  Specifically, my team, the client team, is hiring.  And since Mozy decided to make me the manager of the Windows client team, that means I’m participating in the interviews.  This is stressing me out, because I feel like I am deciding the fate of people.

The process of getting hired at Mozy (arrgh, Decho) goes something like this:

• Apply and submit a resume.
• If we like your resume we will do a phone interview.
• If you do well in the phone interview we will bring you in for on-site interviews.
• If you do well in the on-site interviews we will assign you a homework assignment.
• If you do well on the homework and you are the best candidate we have for the position, there’s a decent chance you’ll get an offer.

Development work at Mozy is primarily done in C++.  Objective-C on the Mac side.  Pretty much you have to know C and C++ and/or Objective-C to get a development job, unless you want to work for the web team, using Ruby.  But those guys are kinda weird.  They sit on a different floor in the building and everything.  We’re not talking about those guys.

So during the phone screen, we ask you to rate yourself on C++.

We explain the rating scale like this:  0 means you are my father, waiting for this computer fad to go away, and you haven’t really heard of C++.  1 means you wrote Hello World in C++ once, and might be able to do it again today.  On the other end of the scale, 10 means your name is Bjarne Stroustrup, or maybe Herb Sutter or Andrei Alexandrescu.  9 means you have written books on C++; 8 means you could write a book on it, or teach courses on it.

Please, people.  Do not flatter yourself on the C++ scale.

I interviewed with Google once, over the phone.  They asked me this same question with pretty much the same scale, except they made no mention of my father.  I told them I was a 6 or a 7.  And I actually have taught courses on C++.

Lately we’re asking people this question and invariably we’re getting people saying, “Oh, based on that scale, I’m a 7 or an 8.”  Even kids in college.  Now I’m not saying that kids in college can’t be a 7 or an 8 – just, keep in mind, we’re not seeing a lot of true 7′s or 8′s among experienced professionals.  I’m just sayin’.

When you say in your interview, “I’m a 7 or an 8,” what you are telling me is this:  “I know C++ better than you.”  Now, you don’t probably know me personally, so hey, maybe you are better.  All I’m saying is, you’d better be ready to prove it when we bring you on-site.

For example, you’d better know at least most of this stuff:

• How to define a template class
• How to correctly define the assignment operator for a class
• How to overload the insertion and extraction operators for a class you define
• How to iterate over an STL vector
• Whether ++i is better, worse, or the same as i++, performance-wise, and why
• What methods the compiler will create for you if you don’t create them yourself, and the implications of this
• How to indicate in your developer contract whether a class is meant to be subclassed, which methods are overrideable, and how you insist that only subclasses can be instantiated
• How to specify default values for parameters

If you are a 7 or an 8, you probably should have read most of “The C++ Programming Language” and/or “Effective C++” and/or “Advanced C++” and/or a number of equivalents.  Having read “Design Patterns” would certainly help, although lately those have kinda lost their glimmer and so I don’t weigh on those like I used to.

Also:

• What const and mutable mean

Yeah.  const.  Don’t be like the self-proclaimed C++ expert I worked with at Enterasys Networks, who told the whole company he was the go-to guy for C++ questions, who, when asked, “Why does it say const after this method declaration?” replied, “Oh, they just do that a lot in C++; it doesn’t mean anything.”  Yeah.  Don’t be a doofus.

Don’t try to impress me by saying you are a 7 or an 8 if you aren’t.  Really – you don’t have to be a 7 or an 8 to get a job at Mozy (Decho…hrm).  If you say, “Oh, I’m probably a 5,” that tells me you are a good, solid C++ dude (or dudette, whatever) that knows how to write decent C++ applications.  You’ll probably get asked to come in for an interview anyway.

When we bring you in, it is my job (and Cody’s) to figure out how much of C++ you really know.  We will start out at the point you specified and go from there.  If you are really a 5 or a 6, but you said 7 or 8, you will feel like we’re being very brutal on you.  Hey, you are the one who said you knew your stuff.

Oh – one more thing.  Some of you experienced hires don’t think you should have to go through all of this to get a job with us.  Well, we make the rules.  Every one of us that works there has gone through it.  If you think the rules of Monopoly are dumb, nobody’s gonna think bad of you if you decide not to play.  But if you want to play but not follow the rules, well, don’t be too surprised if people take issue with that.  If you’re gonna try to work at Moz – uh, Decho, at least for my team, just go through the process like everyone else.

Okay, I feel better.  Whew.  Oh, and by the way, if you really are a 7 or an 8 (or better), I have a link for you.

Suppose you have a very common multiprocessing scenario:  two threads, one of which is producing work to be done, and another which is consuming the work to be done and performing the work.  You can accomplish this in a pretty straightforward fashion with:

• A queue
• Mutexes to prevent concurrent queue access
• A condition variable to tell the consumer when there is work to do

Originally I thought about creating my own container class for this.  Then part of my brain resisted.  “Quiet, engineering part of Matt’s brain!” it exclaimed.  “You’re just trying to do that because it seems Interesting!”  So I shut down that idea and pushed on trying to use a standard STL queue.  However, after I had to deal with some concurrency issues over and over, I realized that, at least this time, the engineering part of my brain was right.

So I set out to create a threadsafe queue.  The idea was that it would feel a lot like an STL queue, and would implement thread safety under the covers for you – so, for example, a simple call to q.empty() would automatically lock the mutex for you, see if the underlying queue was empty, then unlock and return the result.  So, essentially an STL queue wrapper class with automatic mutex locking, right?  It should be easy!

Well, you’d think so, until you get to an implementation of pop().

To see why this matters, first we should establish what we expect out of a threadsafe queue in C++:

• It should feel like an STL queue.  This means that it should have similar semantics and API if possible, and also, it should be a template.
• You get out what you put in.  Not a handle to what you put in; exactly what you put in is what you get out.
• Thread safety should be handled automatically – the user shouldn’t have to deal with thread safety.

An STL queue “pop()” isn’t as simple as it sounds.  The concept of “pop” is actually performed like this:

    std::queue q;
    ...
    if (! q.empty()) {
        int v = q.front();
        q.pop();
    }

Popping an empty queue has strange consequences; in fact, on my machine, it segfaults.  You’d think the queue would be resilient to that behavior but it isn’t.  Also, you have to get the item off the front in a separate operation from actually removing the item.  This is so you can store anything in the queue – pointers, reference objects, etc. – and obtain the item in the front of the queue and know it will survive long enough for you to do something with it.  You pop() the queue after you are finished with the item.

That’s all fine but it makes our threadsafe queue a bit of a problem.  Suppose we decide to follow strictly the STL queue API.  In that case, we’d perform an “if not q.empty(), then q.front() and q.pop()” dance.  We have to assume that each of those three actions is legal on its own though, and we surely can’t release the lock after each one.  So that would mean that the user has to obtain a lock on their own, to be used outside of the whole three-step dance, which breaks one of our rules, namely, that the user shouldn’t have to deal with thread safety.

Okay, so what if we abandon the need for strict adherence to the STL queue semantics, and instead just implement a pop() method that does everything?  Now we can hide the thread safety management from the user.  But now we have a different problem.  How do we know whether the value we get back from pop() is a legal value?  How do we know whether it came from the queue or whether the queue was empty and there was nothing to pop()?  We need some way to check to see whether the value coming back is valid, i.e., whether there was something in the queue to pop beforehand.

There’s a couple of ways to do this, but none I like.

• If the return value were a pointer (or an iterator), we could check for NULL (or end()).  However, we don’t necessarily know that.  If the class is a template, and if we always get back what we put in, we can’t know whether the return value is a pointer or a value or a reference.  True, we could cast the return value to a pointer type – but then, not only are we not returning what we put in, but we are getting into a rather scary situation where the user is required to massage the data before it can be used properly.  And we lose type safety.
• We could return a std::pair<bool, t> where t is the type used to populate the queue.  Not as scary as the casting solution before, but still with the problem where the user has to massage the data, and the return value isn’t strictly what was put in (although it is contained in the pair).
• We could throw an exception if the container is empty, but an empty container isn’t exactly an exceptional case and is probably not a good ethical candidate for throwing an exception.
• We could stop using the STL queue and create our own queue with more reasonable semantics.  However, although I’m not necessarily in love with the STL queue, I’m also not convinced I can do it better, at least not convinced enough that I want to spend the time to do something that already exists.
• We could do away with making the wrapper class a template and instead make it implementation-specific.

Since I don’t know for sure that I will need my threadsafe queue for anything other than a very specific data type I have in mind right now, I chose the last option.  This is the agile way – I’ll meet today’s requirements today, and trust that I can refactor the code to meet future requirements when they show up.

So now everything works great, except doing the wait on the condition variable, which normally looks something like this:

    mutex m;
    condvar cv;
    m.lock();
    if (q.empty()) cv.wait(m);
    v = q.pop();

Yeah, here I am, managing thread safety outside of the queue again.  I really should put this into my threadsafe queue also; except I would need a method something like waitThenDo(action) where action is a function pointer or some other callable.  I hate how that makes my code look though.  And at this point, with the waiting done as shown above (roughly), my code is working, so I’ve stopped caring.

But if you have a great solution to this problem, I’d love to hear it.