A Guide to Hiring Programmers: The High Cost of Low Quality

I was invited to a wonderful dinner party (I swear it wasn't too spicy Sarah!) with some St. Louis Perl peoples this week while I'm here on business.  At one point we were talking about hiring programmers, specifically Perl programmers.

We agreed on the following:

• Finding good programmers is hard in any language.  And that a good programmer can be as effective as 5-10 average programmers.
• Average pay rates between equivalent programmers are out of sync and are based more on the language used than the skill of the programmer.
• You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks.
• You should seriously consider allowing your expert developers to telecommute full-time. Restricting your search to programmers who live in your area or are willing to move limits the talent you can acquire. Arguments regarding "face time", productivity, etc. can easily be nullified when you look at how some of the largest and most successful Open Source projects such as Linux, Apache, and Firefox are developed by individuals rarely living in the same time zone or even country.
• We love Perl and think it's a great language that you graduate to after you have been forced to use less agile languages such as Java, C/C++/C#, etc. Not necessarily a first language you get your feet wet with and then move onto a *cough* "real" language.

Many people in the Perl community have been writing on this topic lately and wanted to share my opinions on the subject, as it is one I have put many hours of thought into. Doing my best to keep this language agnostic as I believe these tips can be applied to any programming language. I will however, use Perl in some examples as it is my preferred language.

Why is it so hard to find good programmers?

The simplest reason is when a company finds a good developer they do more to make sure that person is happy which leads to longer tenures. Better salary, more flexible working conditions, good tools, interesting projects, and better perks can often keep a programmer working for you longer.

Another obvious reason is that experts in any field are small in number, so your possible talent pool is limited. This leads managers and HR departments to settle for average or even below average developers.  I believe this is the single biggest mistake a technology oriented company can make, regarding developers, just short of not using a good version control system.

We're not talking about customer service representatives or sales people here. Just having a body to fill the seat is not, I repeat not, always a win for the company. Sub-standard programmers drag down the efficiency of your other developers with beginner questions, poor comments/documentation, and bad code that someone else will later be forced to spend time fixing.

Companies need to stop thinking about their developers as cogs in the machine. They are more akin to artists, authors, designers, architects, scientists, or CEOs.  Would your HR department rush to find the first person who would be willing to take on the role of Chief Scientist, Art Director, or CEO in your company? Of course not, they would spend the time to do a thorough talent search for just the right candidate, court them, and then compensate them appropriately. They realize that having the wrong person in that seat is much worse than having the seat empty. It is absolutely the same with programming.

Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers.  However, companies typically pay only a 10-20% premium for an expert over the average programmer. Whether or not their title is Lead, Architect, Development Manager, Guru or whatever nomenclature the company uses. I am not saying that if your average developer is paid $50k/year that you should pony up $500k/year for an expert. The employer/employee relationship never works like that, but what employers don't seem to realize is that in the end paying more saves them more.

Let's look at an example.  One common argument from HR departments is that they "can't find any Perl programmers, but they can't swing a cat without hitting a Java developer".  While this is fairly accurate, they are approaching the problem from the wrong direction.  If you fill your shop with 15 average Java developers, paying an average of $60k per developer you have an approximate labor cost of $900k/year for your development staff.  Not considering any non-salary benefits.

Suppose you instead took the time to find 5 experts, or at least above average, Perl developers at $120k each per year. Here is a partial list of the pros and cons of such a scenario:

Cons:

• You must spend extra time finding, evaluating, and courting these more sought after developers.
• Your company and what the developer may be asked to build may simply not be attractive to this class of developer.  Very few people want to work for a spammer or a small web design firm that caters solely to freelance accountants for example. Smart people find boring things even more boring than the masses.
• When one of them leaves the company, there is the feeling that your company's business objectives are more at risk due to having only 4/5ths of your normal resources. Or that a larger chunk of your corporate knowledge just walked out the door. This is more of a perceived problem than an actual one as good developers are better at writing readable/maintainable code, commenting their work, and writing effective documentation.

Pros:

• Each developer will be more content with their job, due in part to the higher than average salary, but also because his or her co-workers are of a much higher quality which improves anyone's job satisfaction.
• Development would require less overall communication as there are less people to communicate with.  This obviously improves efficiency as anyone who has been on a 20+ person conference call can attest to. Or read the Mythical Man Month if you want a more in-depth analysis of this phenomenon.
• Experts travel in the same social circles.  Having one expert on staff makes it much easier to find other experts in the same field, no matter what field that may be.
• You would save 2/3rds on infrastructure costs.  Things like cubicles, computers, cell phones, free lunches, training costs, travel, office space, air conditioning, electricity, etc, etc. The list is essentially endless.
• Your HR department would have 1/3rd the number of developers that it would need to take care of. Less paper work, less questions, less everything, and less turn over because of the lower number of employees.
• Oh and you'd save $300k/year on your labor costs.  Not to mention non-salary benefits such as stock options, retirement matches, health insurance premiums, perks, etc. You could spend as much as $100k/year on your talent searches and still be $200k/year ahead.  Hell, you could dedicate an entire HR person just to this task.

What is an expert programmer?

Experience is key, but not necessarily in ways you might imagine.  Time in the saddle, with a particular language is not as important as diversity of experience.  Someone who has worked in several disparate industries, a generalist, is often a much better developer than one who has spent years in the same industry.  There are exceptions to this, but in general I have found this to be the case.  Bonus points if your developer was a systems administrator in a former life.

Some of the best developers I know were originally trained as journalists, mathmaticians, linguists, and other professions not normally associated with software development.

Experts use better tools and care deeply about their craft.  They aren't assembling bits on an assembly line, they are crafting a unique product to solve a unique problem.  Experts are lazy, they work smarter rather than harder.  Experts prefer the easiest solution that gets the job done. Experts aren't interested in creating complex solutions simply to have the complexity, that misguided egoism is the territory of more junior developers. They often get it right the first try and almost always on the second one.

Simply put, experts write readable code.  They comment and document it appropriately based on the complexity and criticality of that particular piece of code.

All of this pays huge dividends when the next developer has to pick up where they left off. Especially if the next person isn't an expert.

More reasons you want an expert programmer

Is your business technology oriented?  Perhaps the software you create is even your main product. If nothing else I'm sure we can agree that if the software your developers create is to some degree critical to your business.

I've worked in many different environments, with people of every skill level, and it's very easy to tell whether or not a company has expert developers. Do you often find that the software is down? That it has as many bugs or even just idiosyncrasies that make no sense to the user as it does features?  Do the users find it difficult to use?  Is the problem at hand relatively simple compared to the training or documentation necessary to begin using the software?

If you answered yes to any of those questions you more than likely have average or below average developers.

When you work in an environment with experts things simply work.  They are easier to use and require less initial training. The software is easier to modify.  Requested changes happen more frequently and easily.  Things just flow.  It is the difference between Apple and Microsoft.  It's the difference between the iPod and a 400 disc CD changer with 50+ buttons.

As with many things in life, sometimes you get what you pay for. I'd love to hear your comments and opinions on the subject.

UPDATE: I've written a response to some of the questions and comments I've received on this article in a follow up post A follow up to "A Guide for Hiring Programmers"

Comments

[sales guy]

just putting a sales guy in a seat is not a win for the company either. who do you think generates the revenue to allow the company enough funds to hire an 'expert' developer? this was a good write-up, but don't invalidate the effort of the sales team; we make sure the bills get paid.

but we can all hate on hr :-)

[Steve]

Your points are valid, but sometimes money isn't enough to keep good people. I've worked at places where either the work or the corporate culture has been so poor that holding onto good people was incredibly difficult. I've also seen great places to work go sour due to a few poor recruitment choices.

I guess my point is that you can have good people, but that's only part of it. Just like with hiring exports, it helps to take a holistic view to the workplace.

Which, unfortunately, is beyond the power of many .

[lofi]

apart from being grotesquely wrong about Perl being an adequate language, this article is on the right track ;)

[Jimbo]

The problem gets worse as you move up. $120-$200k is the upper cap on engineers at any company (not just software). If you do 10x better work, you get 10% better pay. If you're a hotshot, it is simply not economical to go into industry. You do a startup, contract work, or switch fields (entrepreneurship, i-banking, etc.).

[Frank Wiles]

I agree with all of these comments, except for the Perl one ( that could be a whole article unto itself ). Sorry I singled out sales people, I do realize there are amazing expert sales people out there, it is just in my experience most of them seem to be easily interchangable! :)

[Paul Evans]

"It is the difference between Apple and Microsoft."

Interesting statement.

[Matt C]

Let's say you're right about the cost effectiveness of hiring better programmers.

How is an ordinary company to know the difference between a good and a bad programmer? An HR type who offers 2x pay for 5x the effectiveness is not likely to actually get the 5x. There are plenty of programmers (or "programmers") out there who talk a good interview without delivering much. Being able to tell a genuinely exceptional programmer is a skill in itself, and not a common one.

Also, you are assuming the hiring person is motivated to maximize cost effectiveness, rather than, say, having a large number of people reporting to him so he looks more important. In my experience cost effectiveness is not central to the thinking of manager types.

[Nick Quaranto]

All I see here is more people like me who are new to this industry and who are coming out of college soon being screwed out of good paying jobs. Without 5+ years of experience it seems like I'll be on code monkey status out of college.

[Jarvis]

I found this informative and potentially useful until I read your childish comparison of Apple to Microsoft.
Congratulations on discrediting yourself.

[Dr. Alabaster]

I've walked into a number of failing projects that were started in the latest fad language and witnessed the use of PERL save everyones hides. There aren't many other languages that have the resources of CPAN modules where much of the time, someone's already done the heavy-lifting for you.

PERL/CPAN is a great example of a community sharing their work so other members can turn around and leverage that work in their code. PERL may be "inferior" in a number of ways but at the end of the day, the stuff works, which, I'm finding, bosses seem to like.

[Gunther Hust]

I agree with most of what you have said here, but the main counter issue in my experience is that the people making the decisions can't tell average programmers from expert programmers (or average sales guys from expert sales guys, for that matter). Nor do they care to put the effort into being able to do so - after all it doesn't benefit them.

In a lot of cases, managers do _not_ want great programmers on their team, they are seen as prima donnas, and their technical leadership can encroach upon the authority of the team manager. Great engineers care most about making great products - managers care most about promotions.

The decision makers make decisions mainly based on who they know, not what they know. Getting a team of 5 or 10 average programmers together and delivering something successfully is good for the manager, even though it is not necessarily good for the company. Just like getting a team of 5 to 10 average sales guys together and making 1M in sales is good for the sales manager (whereas maybe the expert sales guy can pull in 5M in one deal).

[kalpesh]

Money alone cannot be motivating factor when it comes to retaining the best talents. Software industry has been plagued by high attrition rates. No company can win the business game with one department even if it means that department is most productive.

[Maxdogg]

"The High Cost of Low Quality" really applies to ALL fields. However high quality does demand a higher cost, just not as high as low quality and most companies are reluctant to spend more than the least possible.
As a chemist I would catch mistakes that my peers were routinely making and cause problem because of "backtracking" to fix the problems.

[rem]

Good Post. The point for job seekers is look for a Company not a Job.

http://jobhacks.wordpress.com/2007/08/01/dont-look-for-a-job-look-for-a-company/

[Peet]

Good points - except that Perl is passable only for the most *cough* simple of applications. For scale, managing complexity, readability, nothing beats Java.

[OneYearGoal.com - $100,000 in one year]

You couldn't be more right. A bad programmer is almost worse than no programmer.

[Andrew Gamache]

I need 10 Outstanding Perl Developers to work for an investment Bank in NYC. We're probably looking at 100K ++. Hopefully some Experience in the Mortgage Business (Application Development for Mortgage Backed Securities- Analytics and High-Speed Trading Systems)

This is going to be the opportuinity of a lifetime for someone.. (It'll certainly make your career) Exected Comp over the Next 5 Years is 1 Million+. No Joke. If you rock at PERL, and have Financial Industry KNowledge/Experience (Or just know alot about Mortgages) email me: agamache@optionsgroup.com
{
Compile your future.
AG
}

[lol @ perl]

We had a perl "programmer", he was very fond of the game perl "golf" and tried to play it with our valuable assignments. We got rid of his stupid ass before he had much chance to cause trouble, and from now on we only hire programmers skilled in the dot net toolset, and its served us very well.

[David F. Skoll]

So, how do you find expert Perl programmers?? We're always looking!

[JB]

Thank you for this post.

I was brought on my current project (non-Perl) when it was six months overdue. The former programmers were novices at best (more likely incompetent), which meant my job entailed progressive rewrites of their code. And, no, I wasn't being snotty about the code. They had used two (sometimes more) functions instead of arguments. BTW, one of those developers gave his notice as soon as I asked for his source code.

The VP didn't want to move me to the project because I was, "too expensive."

In two years, these guys had produced 400 "units" (purposefully vague) for delivery, or 16/month. In less than two months following the restructuring, I and a budding-expert developer produced more than 100, or 50/month. That's better than 3x previous production.

Yet, somehow, I doubt those other guys cost 32% what I did.

[Note: I wouldn't consider myself "expert," but I have a lot of experience working for a variety of clients, and I do try to do things right the first time.]

[Dave Palmer]

>You don't need to hire an expert in language X, you can and should look for expert programmers
>that are willing to learn language X. An expert can easily cross over from being a novice in
>any language in a matter of a few weeks.

A common notion, but wrong.

Sure, any competent programmer can learn the syntax of a new language in short order, and most of the uber-concepts of programming do not change from language to language. But the *details* of the language take time to learn, and a program is made of details. How do I implement a menu control? How do I listen for mouse events? Is there a maximum string size? Are chars signed by default? Many of these details might be tiny and easily-learned, but there are a LOT of them, and a person who is just learning a new language has to look up most of them. That takes time (and BTW, if you're just learning a language, how do you know whether that example you looked up is well-written or not?). Further, some concepts and details just don't translate into a different language.

Really, the proper learning time to consider is not how long it takes to learn the syntax, but how long it takes to learn a useful subset of whatever standard library the language uses to perform useful tasks. I spent years getting familiar with all the details and gotchas of MFC before I felt confident calling myself an expert. Now, I've been dumped into the Java world, and suddenly, there's an entirely different big, bloated, buggy standard library to learn. AND the new concept of cross-platform issues to deal with.

In MFC, I know how to do lots of useful stuff just off the top of my head. Further, I know where many of the gotchas and pitfalls are, and how to avoid them. I know how to work the IDE. I know none of that for Java, and it will take more than a couple weeks to pick it up, no matter how diligent I am.

It is my considered opinion that no small, agile company on a tight schedule should attempt to write code in any language unless they have at least one bona fide expert in that language on staff, somebody who has spent at a minimum 6-12 months writing almost exclusively in that language (of course, I mean *in addition to* having several years of solid general programming experience).

Anybody who tells you they can learn a new (full-featured) language in a few weeks is either lying, deluded, or unclear on the concept of "learn."

[Ethan]

You have convinced me about the need to hire experts but how do I figure out if someone is an expert or not?

I am going to be hiring a few programmers in the near future but am not a professional programmer myself. I have some knowledge in sys admin and web development but know little to nothing about more complex languages like C, C#, C++, Java, etc.

How should I go about determining if someone is an expert. What kinds of questions should I ask and what kind of responses do I want.

Thanks for your post!

[Mikey]

Hmm, I see a void forming...

How do we get expert developers? It seems to me that all expert developers were once average developers. If not all, then I would say most. I'll admit that there are exceptions, geniuses. There are also those who got to be experts by working hard as average developers to improve their skills. This is where I see a void forming in your article.

I agree that an expert work force is ideal. I am saying that that ideal isn't really feasible, because it leads to elite-ism. I would say that the ideal is to have a few experts and several very eager average developers. People who will willingly submit to constructive criticism and learn to better discipline themselves, so that they can become expert developers. That road is messier, but it also leads to more expert developers.

If we are to ever meet your ideal of an expert workforce I think it must be by following this road. I say this because I know that there is only so much time that a person will devote to full time education before seeking real world experience. I think we are lucky to have a few CS Masters graduates that are expert developers.

Perhaps I am a little bias on this last point. I am a Systems Administration Bachelor graduate myself, who has been tasked with a few programming projects. I consider myself a fairly eager amateur, and had I tuned by education to software engineering I might go as far as to call myself average. If I am now an average developer it is only because I am so eager, and have been trying to teach myself the software engineering skills that you have in part described in this article.

To summarize, I think that society can only effectively handle putting people in to so much full time education. There has to be a point where real world experience becomes that teacher, along with the tutelage of the experts. However, this sort of path of advancement is circumvented by what I see as an elitist mentality of only hiring the experts.

[Міша]

I think, this applies just the same to many other professions. Sysadmins for sure. Probably -- other engineers too...

[Susan M]

AMEN - I quit progamming after 20 years not because my skills went stale but because my experience and expertise were not being valued. It seemed that additional experience was not rewarded. Companies would hire 5 junior programmers instead of 2 senior programmers. I predicted that this would come to a crisis in code someday. This article tells me that it is about to.