Revolution Systems Blog

Technique for improving code over time

I was talking with a friend of mine recently about an all too common problem in software development.  That problem is finding the time and resources to revisit working, but not perfect code.  For whatever your definition of "perfect" may be.

The developers feel this particular bit of code is just plain nasty. Maybe it is written in a confusing manner, poorly documented, or the landscape has changed so much that the entire architecture could use a rethink. The programmers may have thought up a better way, changed their style/best practices, or perhaps have just improved their skills since that code was written. But it works. It accomplishes the business goals.

Because it works, the management/marketing/sales folks are not very interested in making it better.  They would rather the programmers focus their efforts on new features, because they don't feel they can sell code quality. "Maybe next month/year/release we can do that, but right now go knock out some of these TODO list items...", is what you typically hear.

Both sides of this struggle need to realize that they both are correct and both incorrect. The developers are right in saying the code isn't perfect and could be improved.  They are also wrong, because sometimes what you have is "good enough".

The biz folks are correct in their thinking that they can't directly sell the results of the programmer's effort. They can't put it on brochure and it certainly won't get them talked about in FastCompany. But they are also wrong, because they are missing the long term impact the code quality/design will have on their future business. It is difficult for them to understand how changing, for example to a messaging oriented architecture, helps their modularity and scalability. How it might drastically reduce future development time, improve testing, etc, etc, etc.

Arguing this out could take forever. Not to mention you will have to argue each architecture change as you find/want them.

A technique I have used with great success in the past is what I have so cleverly named "Friday Afternoons".  Friday afternoons are probably the least productive part of an employee's week.  They are watching the clock in anticipation of the weekend. It happens to everyone.  Experienced sales people will tell you that it is basically impossible for them to get any real sales on Monday mornings and Friday afternoons.

As a compromise,  I suggest management give the developers free reign to make architecture, design, and code clean up changes on Friday afternoons. You may need to implement rules around this, such as the changes can't impact any deadlines, they can't create any new work for others, can't impact the product in adverse ways from the customer's perspective, all work must be done in a cleanup branch, etc.

Some of you managers out there are probably thinking, but "No way, that's 10% of our development time".  If you feel this way, you're really deluding yourself. No one is 100% productive, and I would estimate no one is even 25% productive on Friday afternoons, yourself included.

So why do this? There are several reasons:

• Developers can cleanup the code, effectively making their workplace ( aka the code ) more enjoyable. And everyone knows that Great Code comes from Happy Coders.
• The programmers feel their concerns are being heard. That management cares about about the quality and not just the dollar signs.
• You eliminate the time drain of the meetings arguing about these issues.
• The coders are self-motivated in their Friday afternoon sessions. You weren't going to get much out of them anyway, but at least there is a chance that you can reap future benefits from this work.

Maybe one of those architecture changes means that a requested new feature takes 2 days to implement rather than 2 weeks.  Or perhaps the code will run 5% faster or scale 2x as well.  Even if you just end up with better documentation and a slighly improved employee morale this is win.

If you end up using this technique for awhile, please send me an E-mail or leave a comment here.  I would love to see how this idea plays out in different environments.

I'm in the top 10....

Based on this this article at PerlBuzz.com , my humble little blog is in the Top 10 of all Perl related blogs.  While the author admits to having used a very simple/naive ranking system, it is still an honor to be listed.

For anyone who wants to subscribe to only my Perl related posts you'll want to use this RSS link. You can use the RSS feed link on the right to receive all of my posts. 

I guess I better start writing more about Perl!

Followup to "A Guide to Hiring Programmers"

Please excuse my laziness, but I simply don't have time to respond to each and every person who has E-mailed or left comments on digg, reddit, or the original post itself.  I would like to respond to a few of the larger themes I've seen in the questions/responses:

This applies to more than just programming

I definitely agree that this can be applied to nearly any type of job, not just programming.  A great designer is worth much more than an average one.  And I honestly wasn't trying to single out sales and customer service people.  I do agree that a great sales person or customer service rep is worth more than the average, and should be paid accordingly. And yes every employee is important to the company.

Using customer service as an example, I've worked with the worst where they would literally scream at the customer on the phone to the best. The problem with customer service is that the metrics are against them in that even the best customer service person can only take a few more calls/tickets than the worst.  Just because of the nature of the interactions. It is also very difficult to measure if this person has pleased/retained more customers than another. With programming it is often easier to see how an individuals contributions impact the whole project.

Sales is a different monster, but hopefully this has cleared up what I was trying to express.

Only experts?

Do I think it's reasonable to hire only experts?  Yes, in many situations a company can and should staff themselves with the vast majority being experts. Is it possible for larger companies with larger products? Probably not.  If the problem simply demands 50 developers, it would be difficult to staff that entirely with experts. However, I do believe they would see a boost if they were able to have at least 10-15 of those developers be experts. Instead most companies have 1-3 experts that lead the team of the masses.

If you can't find experts, you should attempt to hire staff that could become experts over time as they gain experience.

How do you become an expert?

Everyone is correct in saying that experts started out as novices, I was certainly a novice.  In many ways I still am.  Being personally interested in martial arts I remember a story of someone, after years of training, finally receiving their black belt in Aikido and being told, "Now you are ready to learn."  I believe this is true of programming, technology, and most professions.  The learning doesn't and shouldn't stop.

So how can you become an expert? I think the best advice I can give is to read up as much as possible on your field.  You don't become an expert simply repeating what you did yesterday for many years until, poof, you're an expert. You need to be learning new idioms, patterns, and tips from your peers.

Too many developers sit in their cubes and pound out code and never look up. You need to be reading up on your profession as much as possible, exploring new languages/tools to determine if you could be doing something easier or better.

An example of what I see far too often happened again recently at OSCON. A professional Python developer did not know Django was the predominant web framework for that language. I'm not a Python user, but even I know this. Maybe it's because I'm friends with the core Django team, but even if that had not been the case I would at least be aware of it and in general what it was from my day to day  tech reading.

The other advice I would give is to read and become involved in an Open Source project.  This improves your code quality and allows you to see how other, presumably senior, developers work. Even if you aren't able to contribute to the project directly, get on the mailing lists and examine how those developers work.

How do you find and hire experts?

I think the biggest mistake managers make is leaving this up to HR.  I've always made sure I received every resume that came in for a position I was hiring for.  HR will often reject a candidate because their resume states they have "Years of J2EE experience", but since it's a Java programming position it goes in the trash.  Perhaps it is time we start hiring "HR Engineers" like we have "Sales Engineers."

The first place I look when hiring programmers is the Open Source community. If they are involved in an Open Source project you can easily see how they work with others on the mailing lists, see examples of their code, etc.

They also tend to be of higher quality because Open Source is a meritocracy. Not to mention the simple act of being involved in a project, for no monetary gain, shows a strong love of their craft.

I think multiple choice tests are a very poor indicator of programming prowess. Too often they have a couple of esoteric or even trick questions that really compare the test writers ability to confuse with the test takers' ability to decipher.  It is much more important for your new hire to know how to find the answer than it is for him to actually have it tucked away in a brain cell. Ability to effectively use Google to search for the answer is much more important than many realize.

If you happen to be one of the people who are looking for an expert Perl programmer I suggest you get in touch with my new friend Uri Guttman, The Perl Hunter, at uri@perloncall.com.  He specializes in finding execellent Perl programmers for companies. Being an accomplished developer himself he easily separates the wheat from the chaff and can find someone who will be a good overall fit for your organization.

Many problems are marketing and management's fault...

This is also very true. Bad management will bring down any team or project, no matter how many experts they have on staff. This isn't even restricted to technology management.

Marketing often over promises what can be delivered and demands it in an unreasonable time frame.  Unfortunately most of the time we blame the developers, because long after the sale all that we see is the code and not the brochure.

My advice to marketing and management is that you bring a problem to your developers and then base your plans on when they believe they can deliver the solution.  Far too often management has already determined time lines and set things in motion before the development team has even been told about the project.  This is backwards.  You don't schedule your building contractors before you have the proper permits or before even speaking with the architect about the project.

Even Microsoft gets this right. They realized it was much better to delay Vista until it is ready than to ship it too early just because they originally said a certain date.

Obviously you can't always just wait around for something to be perfect. There are always restrictions and requirements that are outside of your control.  No one could move January 1st, 2000 out a few more weeks just because their Y2K cleanup wasn't done. But often I see companies attempt to move mountains to hit some arbitrary date when one of the largest consequences of delaying would be everyone had to update their Outlook calendars.

My language bias

I received a bunch of comments on my use of "Perl vs Java" in the example, that simply was what we were talking about at dinner that night. I probably should have used "agile language X vs cumbersome language Y" to keep the flames down to a minimum.

You can write efficient, readable, and maintainable Perl.  I've even had some notable Python programmers say that about code I was in charge of and honestly the code in question wasn't what I would consider the best of the best. I think Python is a great language, but for me personally I haven't been shown any compelling reason to switch.

You can write crappy unreadable code in any language. You can make most any language/framework/toolset scale and perform to your needs.  For every "large app/website/etc" that uses language X I'm sure I can find you a comparable app/website using language Y.  Any performance differences between language X and Y can usually be solved with $100 worth of extra CPU.  What really matters is programmer efficiency.  That is where you save money and reap benefits. I simply don't see how having to write, read, troubleshoot, and maintain 10x the number of lines of code is an efficient use of the programmer's time.

However, I do agree that you should use the right tool for the right job. Java/C++/C# are definitely the right tools in many situations.  I just feel that because everyone has seen a horribly written Perl CGI ( or written one themselves ) they think this is somehow ingrained in the language and because of this Perl simply isn't an option for anything "real."

Perl is a language where the developer must use some self-control rather than having it imposed on him by his tool. Which is why Perl (or many of the more agile languages such as PHP/Python/etc) written by novice programmers is so awful. The knowledge and self-control comes with experience.

The largest problem with any language is the use of poor variable, function, class, and method names.  Using adequately long and descriptive names is probably the single best way to improve code quality and no language out there enforces this. Some enforce a certain style, others force certain methodologies, but this is really only picking at less important aspects of the problem.

Company bias

By comparing Apple vs Microsoft I wasn't really singling out their development staffs.  I'm sure their management, design, and marketing departments are as much to blame for any successes or failures these companies have.

What I was trying to get across was the "It simply works."  I would say the second most common comment I hear from Mac users, after how pretty/well designed they are, is that it "just works."  I don't hear that very often from Microsoft users.

A Guide to Hiring Programmers: The High Cost of Low Quality

I was invited to a wonderful dinner party (I swear it wasn't too spicy Sarah!) with some St. Louis Perl peoples this week while I'm here on business.  At one point we were talking about hiring programmers, specifically Perl programmers.

We agreed on the following:

• Finding good programmers is hard in any language.  And that a good programmer can be as effective as 5-10 average programmers.
• Average pay rates between equivalent programmers are out of sync and are based more on the language used than the skill of the programmer.
• You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks.
• You should seriously consider allowing your expert developers to telecommute full-time. Restricting your search to programmers who live in your area or are willing to move limits the talent you can acquire. Arguments regarding "face time", productivity, etc. can easily be nullified when you look at how some of the largest and most successful Open Source projects such as Linux, Apache, and Firefox are developed by individuals rarely living in the same time zone or even country.
• We love Perl and think it's a great language that you graduate to after you have been forced to use less agile languages such as Java, C/C++/C#, etc. Not necessarily a first language you get your feet wet with and then move onto a *cough* "real" language.

Many people in the Perl community have been writing on this topic lately and wanted to share my opinions on the subject, as it is one I have put many hours of thought into. Doing my best to keep this language agnostic as I believe these tips can be applied to any programming language. I will however, use Perl in some examples as it is my preferred language.

Why is it so hard to find good programmers?

The simplest reason is when a company finds a good developer they do more to make sure that person is happy which leads to longer tenures. Better salary, more flexible working conditions, good tools, interesting projects, and better perks can often keep a programmer working for you longer.

Another obvious reason is that experts in any field are small in number, so your possible talent pool is limited. This leads managers and HR departments to settle for average or even below average developers.  I believe this is the single biggest mistake a technology oriented company can make, regarding developers, just short of not using a good version control system.

We're not talking about customer service representatives or sales people here. Just having a body to fill the seat is not, I repeat not, always a win for the company. Sub-standard programmers drag down the efficiency of your other developers with beginner questions, poor comments/documentation, and bad code that someone else will later be forced to spend time fixing.

Companies need to stop thinking about their developers as cogs in the machine. They are more akin to artists, authors, designers, architects, scientists, or CEOs.  Would your HR department rush to find the first person who would be willing to take on the role of Chief Scientist, Art Director, or CEO in your company? Of course not, they would spend the time to do a thorough talent search for just the right candidate, court them, and then compensate them appropriately. They realize that having the wrong person in that seat is much worse than having the seat empty. It is absolutely the same with programming.

Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers.  However, companies typically pay only a 10-20% premium for an expert over the average programmer. Whether or not their title is Lead, Architect, Development Manager, Guru or whatever nomenclature the company uses. I am not saying that if your average developer is paid $50k/year that you should pony up $500k/year for an expert. The employer/employee relationship never works like that, but what employers don't seem to realize is that in the end paying more saves them more.

Let's look at an example.  One common argument from HR departments is that they "can't find any Perl programmers, but they can't swing a cat without hitting a Java developer".  While this is fairly accurate, they are approaching the problem from the wrong direction.  If you fill your shop with 15 average Java developers, paying an average of $60k per developer you have an approximate labor cost of $900k/year for your development staff.  Not considering any non-salary benefits.

Suppose you instead took the time to find 5 experts, or at least above average, Perl developers at $120k each per year. Here is a partial list of the pros and cons of such a scenario:

Cons:

• You must spend extra time finding, evaluating, and courting these more sought after developers.
• Your company and what the developer may be asked to build may simply not be attractive to this class of developer.  Very few people want to work for a spammer or a small web design firm that caters solely to freelance accountants for example. Smart people find boring things even more boring than the masses.
• When one of them leaves the company, there is the feeling that your company's business objectives are more at risk due to having only 4/5ths of your normal resources. Or that a larger chunk of your corporate knowledge just walked out the door. This is more of a perceived problem than an actual one as good developers are better at writing readable/maintainable code, commenting their work, and writing effective documentation.

Pros:

• Each developer will be more content with their job, due in part to the higher than average salary, but also because his or her co-workers are of a much higher quality which improves anyone's job satisfaction.
• Development would require less overall communication as there are less people to communicate with.  This obviously improves efficiency as anyone who has been on a 20+ person conference call can attest to. Or read the Mythical Man Month if you want a more in-depth analysis of this phenomenon.
• Experts travel in the same social circles.  Having one expert on staff makes it much easier to find other experts in the same field, no matter what field that may be.
• You would save 2/3rds on infrastructure costs.  Things like cubicles, computers, cell phones, free lunches, training costs, travel, office space, air conditioning, electricity, etc, etc. The list is essentially endless.
• Your HR department would have 1/3rd the number of developers that it would need to take care of. Less paper work, less questions, less everything, and less turn over because of the lower number of employees.
• Oh and you'd save $300k/year on your labor costs.  Not to mention non-salary benefits such as stock options, retirement matches, health insurance premiums, perks, etc. You could spend as much as $100k/year on your talent searches and still be $200k/year ahead.  Hell, you could dedicate an entire HR person just to this task.

What is an expert programmer?

Experience is key, but not necessarily in ways you might imagine.  Time in the saddle, with a particular language is not as important as diversity of experience.  Someone who has worked in several disparate industries, a generalist, is often a much better developer than one who has spent years in the same industry.  There are exceptions to this, but in general I have found this to be the case.  Bonus points if your developer was a systems administrator in a former life.

Some of the best developers I know were originally trained as journalists, mathmaticians, linguists, and other professions not normally associated with software development.

Experts use better tools and care deeply about their craft.  They aren't assembling bits on an assembly line, they are crafting a unique product to solve a unique problem.  Experts are lazy, they work smarter rather than harder.  Experts prefer the easiest solution that gets the job done. Experts aren't interested in creating complex solutions simply to have the complexity, that misguided egoism is the territory of more junior developers. They often get it right the first try and almost always on the second one.

Simply put, experts write readable code.  They comment and document it appropriately based on the complexity and criticality of that particular piece of code.

All of this pays huge dividends when the next developer has to pick up where they left off. Especially if the next person isn't an expert.

More reasons you want an expert programmer

Is your business technology oriented?  Perhaps the software you create is even your main product. If nothing else I'm sure we can agree that if the software your developers create is to some degree critical to your business.

I've worked in many different environments, with people of every skill level, and it's very easy to tell whether or not a company has expert developers. Do you often find that the software is down? That it has as many bugs or even just idiosyncrasies that make no sense to the user as it does features?  Do the users find it difficult to use?  Is the problem at hand relatively simple compared to the training or documentation necessary to begin using the software?

If you answered yes to any of those questions you more than likely have average or below average developers.

When you work in an environment with experts things simply work.  They are easier to use and require less initial training. The software is easier to modify.  Requested changes happen more frequently and easily.  Things just flow.  It is the difference between Apple and Microsoft.  It's the difference between the iPod and a 400 disc CD changer with 50+ buttons.

As with many things in life, sometimes you get what you pay for. I'd love to hear your comments and opinions on the subject.

UPDATE: I've written a response to some of the questions and comments I've received on this article in a follow up post A follow up to "A Guide for Hiring Programmers"