Revolution Systems Blog

Making the software written in any language more readable

There are two very simple ways to improve the readability and maintenance of the software you write. They are so simple they are often ignored in favor of more complicated tools and the various programming methodologies people blather on about.  This comes from our human nature to think our own problems are more special and complicated than they really are and from not following the KISS principle.

So how do you improve your software?

By using better names and being consistent. It really is that simple, which is probably why it is overlooked.  A development manager might score some points with his boss by switching to Agile programming, having some scrums, doubling the amount of developer documentation or maybe even switching to a whole new platform like Ruby on Rails.  But who scores any points by saying, "We're going to do better about naming things appropriately?"

Appropriate Naming

Obviously having a variable named temp_user isn't all that descriptive, but it is an improvement over just temp or simply t. Most, if not all, programmers realize this.  However, you will often see variables named clt when they should really be named client, as if those three extra characters were single handedly going to cause carpal tunnel.  Your variable names should be as descriptive as possible without being absurd about it.  A variable named the_current_user_object_we_are_working_with is obviously overkill. But perhaps current_user_object or even current_user is appropriate.

Naming your functions and methods should also be given the same amount of care. Naming a function fix_client doesn't tell us anything useful, we can only speculate something is wrong with the client or the data and this function does something about it.  normalize_client_name is a much better name when you read that all the function does is properly set the case of the letters in the client's name.

The names you choose for your libraries are also very important.  This is one of the reasons programmers find Perl's CPAN much more useful than other programming languages' library collections. Things are named and categorized (for the most part) sanely.  Need something related to E-mail, check the libraries in the Mail category.  Can you guess what Net::SSH, IO::File::CompressOnClose, and WWW::Google::News do? Yeah I thought you could.

If I stumble upon your use of a library called Util, it doesn't tell me anything I still have to go look at the library to see how it fits in with this code.  If it had been named something like  DB::Util or Client::Utils I would at least know the library is probably related to the database or client.

Consistency

Consistency is another area where you can improve your code base without much effort.  If all of your classes contain an initialization method, they should all be named the same.  Not initialize in some classes and init in others.  Things that should be consistent throughout your code base, not only consistent within a single application. If possible, the source in your department should be consistent with other departments and business units.

Consistency through conventions is one of the main reasons people like web frameworks like Ruby on Rails. I'm not advocating Ruby on Rails necessarily, you can accomplish these same things in any language.

Things that should be consistent:

• variable, function, and method naming conventions ( underscores or CamelCase, but not both )
• frequency and layout of comments
• documentation
• configuration file syntax
• on disk layout of source code, binaries, configuration files, etc.
• installation, upgrades and package management
• language(s) used for development

The point being the more consistent you are in how you build an application the easier it is to get down to the task at hand.  Be it new development or bug fixing.  Got a configuration file format you like and have already written libraries to parse it? Then use it EVERYWHERE, in every single darn application you build. No one has to learn the new format to start developing, no time is spent discussing which format is better, and another developer in your organization can jump in to fix bugs or add a new feature.

The last development shop I ran we focused on being consistent. 95% of the time we were building web applications that were either used internally by fellow employees or externally by our customers.  If you saw the source to one of our applications, then all of our other applications would seem very familiar. One might be a ticket tracking system and the other an accounting package, but the layout, coding style, and use of common libraries let the developers dive right in and not have to worry so much about how this particular app is written differently than the others.

Consistency in your applications also makes refactoring easier.  If all of your applications use a particular technique, library, etc. in the same way replacing it with a new tool you have fallen in love with is much easier.  If everyone is doing everything even slightly differently, you have to start worrying more and more how the change might impact your code.

I'm definitely a "right tool for the right job" sort of fellow, but mixing and matching tools and techniques for every single application you build is a recipe for disaster. One shop I know (name withheld to protect the guilty) has two developers.  One who works entirely in Perl, the other entirely in Ruby.  Another application I saw was written mostly in Java, but with a smattering of C# and Python around the edges for kicks. None of these three were chosen from a "right tool for the job" perspective, but simply because those were the favorite languages of the specific developers tasked with those sub-systems. These are obviously worst case examples.

These ideas aren't new, I'm positive I am not the first to use them or even write about them.  But I see these two simple rules violated so often by programmers of all experience levels that I felt the need to reiterate them.

Under-used CPAN Modules

Perl often suffers from its history.  While it is great that there are so many online tutorials and code samples out there for new Perl developers to learn from, they sometimes miss out on all of the useful new code that has been created since the article was written.  Here are a few CPAN modules that are, in my opinion, under-used by new developers.

These modules are great for several reasons.  The most compelling ones are there is lesss code for you to write/debug than not using these modules, the resuling code is often clearer to your fellow developers, and in many cases the module will out perform your hand hacked code.

List manipulations

There are two list related modules you should familiarize yourself with.  They are List::Util and it's cousin List::MoreUtils.

Need to shuffle the values in an array? Don't reinvent the wheel, use List::Util's shuffle function.

use strict; 
use warnings; 

use List::Util qw( shuffle ); 

my @array = qw( 1 2 3 4 5 6 7 8 9 ); 
my @shuffled = shuffle( @array ); 

Need to make a list only contain unique values? You can use the method found in the Perl Cookbook, or you can use the simpler uniq() function from List::MoreUtils like so:

use strict; 
use warnings; 

use List::MoreUtils qw( uniq ); 

my @array = qw( 1 1 2 2 3 3 4 4 5 5 ); 
my @unique_values = uniq( @array ); 

Not to mention List::MoreUtils' implementation is done in C, so it is much faster than a Pure Perl implementation.  I once saw this improve the performance of a large web application by 5% because the developers were uniquing several lists for each page view.

Be sure to also check out the other functions in these two modules such as any(), all(), first(), max(), etc. While you will have to install List::MoreUtils from CPAN, List::Util is part of Perl core.

Merging Hashes

While you can use the simple:

my %merged_hash = ( %hash_one, %hash_two ); 

Someitmes you need a bit more power, enter Hash::Merge.  It gives you several options on how to handle conflicting keys, based on a left/right order, or by storage method. Or you can even control whether or not your data is cloned on not.  I find this very useful for merging in command line arguments against a configuration file.

Speaking of configuration options and files....

Repeat after me.  I will not write my own configuration parsing code.  I will not invent my own configuration file format. I will not parse my own command line options without a damn good reason.

Why waste your time on the most boring part of your code? Use one of the already existing modules and configuration file formats.  This saves you time, debugging headaches, and makes the configuration file syntax familiar to your users. 

I strongly suggest looking into Getopt::Long for handling command line arguments. It might take you a bit of time to get used to this module, but once you're over the initial learning curve you'll wonder why you ever bothered doing this by hand in the first place.

If you like Apache style configuration files (who doesn't?), start using Config::General and cut your configuration processing code into a use statement and a couple of lines of code. Or if you happen to prefer .INI style configuration files take a look at Config::Tiny.

I'm in the top 10....

Based on this this article at PerlBuzz.com , my humble little blog is in the Top 10 of all Perl related blogs.  While the author admits to having used a very simple/naive ranking system, it is still an honor to be listed.

For anyone who wants to subscribe to only my Perl related posts you'll want to use this RSS link. You can use the RSS feed link on the right to receive all of my posts. 

I guess I better start writing more about Perl!

Followup to "A Guide to Hiring Programmers"

Please excuse my laziness, but I simply don't have time to respond to each and every person who has E-mailed or left comments on digg, reddit, or the original post itself.  I would like to respond to a few of the larger themes I've seen in the questions/responses:

This applies to more than just programming

I definitely agree that this can be applied to nearly any type of job, not just programming.  A great designer is worth much more than an average one.  And I honestly wasn't trying to single out sales and customer service people.  I do agree that a great sales person or customer service rep is worth more than the average, and should be paid accordingly. And yes every employee is important to the company.

Using customer service as an example, I've worked with the worst where they would literally scream at the customer on the phone to the best. The problem with customer service is that the metrics are against them in that even the best customer service person can only take a few more calls/tickets than the worst.  Just because of the nature of the interactions. It is also very difficult to measure if this person has pleased/retained more customers than another. With programming it is often easier to see how an individuals contributions impact the whole project.

Sales is a different monster, but hopefully this has cleared up what I was trying to express.

Only experts?

Do I think it's reasonable to hire only experts?  Yes, in many situations a company can and should staff themselves with the vast majority being experts. Is it possible for larger companies with larger products? Probably not.  If the problem simply demands 50 developers, it would be difficult to staff that entirely with experts. However, I do believe they would see a boost if they were able to have at least 10-15 of those developers be experts. Instead most companies have 1-3 experts that lead the team of the masses.

If you can't find experts, you should attempt to hire staff that could become experts over time as they gain experience.

How do you become an expert?

Everyone is correct in saying that experts started out as novices, I was certainly a novice.  In many ways I still am.  Being personally interested in martial arts I remember a story of someone, after years of training, finally receiving their black belt in Aikido and being told, "Now you are ready to learn."  I believe this is true of programming, technology, and most professions.  The learning doesn't and shouldn't stop.

So how can you become an expert? I think the best advice I can give is to read up as much as possible on your field.  You don't become an expert simply repeating what you did yesterday for many years until, poof, you're an expert. You need to be learning new idioms, patterns, and tips from your peers.

Too many developers sit in their cubes and pound out code and never look up. You need to be reading up on your profession as much as possible, exploring new languages/tools to determine if you could be doing something easier or better.

An example of what I see far too often happened again recently at OSCON. A professional Python developer did not know Django was the predominant web framework for that language. I'm not a Python user, but even I know this. Maybe it's because I'm friends with the core Django team, but even if that had not been the case I would at least be aware of it and in general what it was from my day to day  tech reading.

The other advice I would give is to read and become involved in an Open Source project.  This improves your code quality and allows you to see how other, presumably senior, developers work. Even if you aren't able to contribute to the project directly, get on the mailing lists and examine how those developers work.

How do you find and hire experts?

I think the biggest mistake managers make is leaving this up to HR.  I've always made sure I received every resume that came in for a position I was hiring for.  HR will often reject a candidate because their resume states they have "Years of J2EE experience", but since it's a Java programming position it goes in the trash.  Perhaps it is time we start hiring "HR Engineers" like we have "Sales Engineers."

The first place I look when hiring programmers is the Open Source community. If they are involved in an Open Source project you can easily see how they work with others on the mailing lists, see examples of their code, etc.

They also tend to be of higher quality because Open Source is a meritocracy. Not to mention the simple act of being involved in a project, for no monetary gain, shows a strong love of their craft.

I think multiple choice tests are a very poor indicator of programming prowess. Too often they have a couple of esoteric or even trick questions that really compare the test writers ability to confuse with the test takers' ability to decipher.  It is much more important for your new hire to know how to find the answer than it is for him to actually have it tucked away in a brain cell. Ability to effectively use Google to search for the answer is much more important than many realize.

If you happen to be one of the people who are looking for an expert Perl programmer I suggest you get in touch with my new friend Uri Guttman, The Perl Hunter, at uri@perloncall.com.  He specializes in finding execellent Perl programmers for companies. Being an accomplished developer himself he easily separates the wheat from the chaff and can find someone who will be a good overall fit for your organization.

Many problems are marketing and management's fault...

This is also very true. Bad management will bring down any team or project, no matter how many experts they have on staff. This isn't even restricted to technology management.

Marketing often over promises what can be delivered and demands it in an unreasonable time frame.  Unfortunately most of the time we blame the developers, because long after the sale all that we see is the code and not the brochure.

My advice to marketing and management is that you bring a problem to your developers and then base your plans on when they believe they can deliver the solution.  Far too often management has already determined time lines and set things in motion before the development team has even been told about the project.  This is backwards.  You don't schedule your building contractors before you have the proper permits or before even speaking with the architect about the project.

Even Microsoft gets this right. They realized it was much better to delay Vista until it is ready than to ship it too early just because they originally said a certain date.

Obviously you can't always just wait around for something to be perfect. There are always restrictions and requirements that are outside of your control.  No one could move January 1st, 2000 out a few more weeks just because their Y2K cleanup wasn't done. But often I see companies attempt to move mountains to hit some arbitrary date when one of the largest consequences of delaying would be everyone had to update their Outlook calendars.

My language bias

I received a bunch of comments on my use of "Perl vs Java" in the example, that simply was what we were talking about at dinner that night. I probably should have used "agile language X vs cumbersome language Y" to keep the flames down to a minimum.

You can write efficient, readable, and maintainable Perl.  I've even had some notable Python programmers say that about code I was in charge of and honestly the code in question wasn't what I would consider the best of the best. I think Python is a great language, but for me personally I haven't been shown any compelling reason to switch.

You can write crappy unreadable code in any language. You can make most any language/framework/toolset scale and perform to your needs.  For every "large app/website/etc" that uses language X I'm sure I can find you a comparable app/website using language Y.  Any performance differences between language X and Y can usually be solved with $100 worth of extra CPU.  What really matters is programmer efficiency.  That is where you save money and reap benefits. I simply don't see how having to write, read, troubleshoot, and maintain 10x the number of lines of code is an efficient use of the programmer's time.

However, I do agree that you should use the right tool for the right job. Java/C++/C# are definitely the right tools in many situations.  I just feel that because everyone has seen a horribly written Perl CGI ( or written one themselves ) they think this is somehow ingrained in the language and because of this Perl simply isn't an option for anything "real."

Perl is a language where the developer must use some self-control rather than having it imposed on him by his tool. Which is why Perl (or many of the more agile languages such as PHP/Python/etc) written by novice programmers is so awful. The knowledge and self-control comes with experience.

The largest problem with any language is the use of poor variable, function, class, and method names.  Using adequately long and descriptive names is probably the single best way to improve code quality and no language out there enforces this. Some enforce a certain style, others force certain methodologies, but this is really only picking at less important aspects of the problem.

Company bias

By comparing Apple vs Microsoft I wasn't really singling out their development staffs.  I'm sure their management, design, and marketing departments are as much to blame for any successes or failures these companies have.

What I was trying to get across was the "It simply works."  I would say the second most common comment I hear from Mac users, after how pretty/well designed they are, is that it "just works."  I don't hear that very often from Microsoft users.

A Guide to Hiring Programmers: The High Cost of Low Quality

I was invited to a wonderful dinner party (I swear it wasn't too spicy Sarah!) with some St. Louis Perl peoples this week while I'm here on business.  At one point we were talking about hiring programmers, specifically Perl programmers.

We agreed on the following:

• Finding good programmers is hard in any language.  And that a good programmer can be as effective as 5-10 average programmers.
• Average pay rates between equivalent programmers are out of sync and are based more on the language used than the skill of the programmer.
• You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks.
• You should seriously consider allowing your expert developers to telecommute full-time. Restricting your search to programmers who live in your area or are willing to move limits the talent you can acquire. Arguments regarding "face time", productivity, etc. can easily be nullified when you look at how some of the largest and most successful Open Source projects such as Linux, Apache, and Firefox are developed by individuals rarely living in the same time zone or even country.
• We love Perl and think it's a great language that you graduate to after you have been forced to use less agile languages such as Java, C/C++/C#, etc. Not necessarily a first language you get your feet wet with and then move onto a *cough* "real" language.

Many people in the Perl community have been writing on this topic lately and wanted to share my opinions on the subject, as it is one I have put many hours of thought into. Doing my best to keep this language agnostic as I believe these tips can be applied to any programming language. I will however, use Perl in some examples as it is my preferred language.

Why is it so hard to find good programmers?

The simplest reason is when a company finds a good developer they do more to make sure that person is happy which leads to longer tenures. Better salary, more flexible working conditions, good tools, interesting projects, and better perks can often keep a programmer working for you longer.

Another obvious reason is that experts in any field are small in number, so your possible talent pool is limited. This leads managers and HR departments to settle for average or even below average developers.  I believe this is the single biggest mistake a technology oriented company can make, regarding developers, just short of not using a good version control system.

We're not talking about customer service representatives or sales people here. Just having a body to fill the seat is not, I repeat not, always a win for the company. Sub-standard programmers drag down the efficiency of your other developers with beginner questions, poor comments/documentation, and bad code that someone else will later be forced to spend time fixing.

Companies need to stop thinking about their developers as cogs in the machine. They are more akin to artists, authors, designers, architects, scientists, or CEOs.  Would your HR department rush to find the first person who would be willing to take on the role of Chief Scientist, Art Director, or CEO in your company? Of course not, they would spend the time to do a thorough talent search for just the right candidate, court them, and then compensate them appropriately. They realize that having the wrong person in that seat is much worse than having the seat empty. It is absolutely the same with programming.

Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers.  However, companies typically pay only a 10-20% premium for an expert over the average programmer. Whether or not their title is Lead, Architect, Development Manager, Guru or whatever nomenclature the company uses. I am not saying that if your average developer is paid $50k/year that you should pony up $500k/year for an expert. The employer/employee relationship never works like that, but what employers don't seem to realize is that in the end paying more saves them more.

Let's look at an example.  One common argument from HR departments is that they "can't find any Perl programmers, but they can't swing a cat without hitting a Java developer".  While this is fairly accurate, they are approaching the problem from the wrong direction.  If you fill your shop with 15 average Java developers, paying an average of $60k per developer you have an approximate labor cost of $900k/year for your development staff.  Not considering any non-salary benefits.

Suppose you instead took the time to find 5 experts, or at least above average, Perl developers at $120k each per year. Here is a partial list of the pros and cons of such a scenario:

Cons:

• You must spend extra time finding, evaluating, and courting these more sought after developers.
• Your company and what the developer may be asked to build may simply not be attractive to this class of developer.  Very few people want to work for a spammer or a small web design firm that caters solely to freelance accountants for example. Smart people find boring things even more boring than the masses.
• When one of them leaves the company, there is the feeling that your company's business objectives are more at risk due to having only 4/5ths of your normal resources. Or that a larger chunk of your corporate knowledge just walked out the door. This is more of a perceived problem than an actual one as good developers are better at writing readable/maintainable code, commenting their work, and writing effective documentation.

Pros:

• Each developer will be more content with their job, due in part to the higher than average salary, but also because his or her co-workers are of a much higher quality which improves anyone's job satisfaction.
• Development would require less overall communication as there are less people to communicate with.  This obviously improves efficiency as anyone who has been on a 20+ person conference call can attest to. Or read the Mythical Man Month if you want a more in-depth analysis of this phenomenon.
• Experts travel in the same social circles.  Having one expert on staff makes it much easier to find other experts in the same field, no matter what field that may be.
• You would save 2/3rds on infrastructure costs.  Things like cubicles, computers, cell phones, free lunches, training costs, travel, office space, air conditioning, electricity, etc, etc. The list is essentially endless.
• Your HR department would have 1/3rd the number of developers that it would need to take care of. Less paper work, less questions, less everything, and less turn over because of the lower number of employees.
• Oh and you'd save $300k/year on your labor costs.  Not to mention non-salary benefits such as stock options, retirement matches, health insurance premiums, perks, etc. You could spend as much as $100k/year on your talent searches and still be $200k/year ahead.  Hell, you could dedicate an entire HR person just to this task.

What is an expert programmer?

Experience is key, but not necessarily in ways you might imagine.  Time in the saddle, with a particular language is not as important as diversity of experience.  Someone who has worked in several disparate industries, a generalist, is often a much better developer than one who has spent years in the same industry.  There are exceptions to this, but in general I have found this to be the case.  Bonus points if your developer was a systems administrator in a former life.

Some of the best developers I know were originally trained as journalists, mathmaticians, linguists, and other professions not normally associated with software development.

Experts use better tools and care deeply about their craft.  They aren't assembling bits on an assembly line, they are crafting a unique product to solve a unique problem.  Experts are lazy, they work smarter rather than harder.  Experts prefer the easiest solution that gets the job done. Experts aren't interested in creating complex solutions simply to have the complexity, that misguided egoism is the territory of more junior developers. They often get it right the first try and almost always on the second one.

Simply put, experts write readable code.  They comment and document it appropriately based on the complexity and criticality of that particular piece of code.

All of this pays huge dividends when the next developer has to pick up where they left off. Especially if the next person isn't an expert.

More reasons you want an expert programmer

Is your business technology oriented?  Perhaps the software you create is even your main product. If nothing else I'm sure we can agree that if the software your developers create is to some degree critical to your business.

I've worked in many different environments, with people of every skill level, and it's very easy to tell whether or not a company has expert developers. Do you often find that the software is down? That it has as many bugs or even just idiosyncrasies that make no sense to the user as it does features?  Do the users find it difficult to use?  Is the problem at hand relatively simple compared to the training or documentation necessary to begin using the software?

If you answered yes to any of those questions you more than likely have average or below average developers.

When you work in an environment with experts things simply work.  They are easier to use and require less initial training. The software is easier to modify.  Requested changes happen more frequently and easily.  Things just flow.  It is the difference between Apple and Microsoft.  It's the difference between the iPod and a 400 disc CD changer with 50+ buttons.

As with many things in life, sometimes you get what you pay for. I'd love to hear your comments and opinions on the subject.

UPDATE: I've written a response to some of the questions and comments I've received on this article in a follow up post A follow up to "A Guide for Hiring Programmers"

SCALE Talk Slides and Audio

I just noticed that the Southern California Linux Expo (SCALE) has my slides and a MP3 of the audio of the talk up on their site.  The audio isn't actually too bad, especially considering I forgot I was being taped and moved away from the microphone during the Q&A section. 

You can also find OpenOffice and PDF versions of the slides on the Revolution Systems site. Enjoy!

Gantry Article

Phil Crow, one of the Gantry  core developers and the creator of Bigtop has a great article on how easy it is to build and modify Gantry applications using Bigtop.

I highly recommend checking it out, it shows off some of Gantry and Bigtop's best features.

Email, Templates, and Perl

I have been meaning to talk about one of my new favorite Perl modules, MIME::Lite::TT::HTML , for quite a while now.  As I mentioned in a previous post, there are a bazillion different ways to send an Email message from Perl.  This one is just my new favorite.

Here is a short list as to why:

• Can be used for complex multi-part messages and handles attachments easily
• Built upon the equally great MIME::Lite module
• Allows you to easily template your messages using the familiar Template Toolkit package

The templating part is, in my opinion, the important part.  How many times have you had to go edit some source code just to change the text or subject of a message?  Isn't that just terribly annoying. We use configuration files, MVC with HTML templates, etc, etc. to not hard code things into our apps, but for some reason many people ( myself included for years ) have neglected Email.

Not any longer, I've switched to using this module as my standard way of sending Email these days.  If you are interested in learning more about MIME::Lite::TT::HTML, check out my short howto Sending Email with Perl Best Practice on the subject.

Gantry book released to the world

Phil Crow has released a new book on using Gantry and Bigtop.  It's titled Building Web Applications with Gantry and Bigtop I encourage you to check it out!

Installing Mail::Cclient on RHEL

Ran into a small build problem when trying to install the CPAN module Mail::Cclient on a Red Hat Enterprise Linux system.  Figured I would go ahead and document the process in full for others ( and for myself later in life ). 

FIrst off you will most likely need to install two RPMs

• libc-client
• libc-client-devel

Then you will need to download a distribution of Mail::Cclient, don't even bother trying to install this from the CPAN shell as it won't work.

Unpack your distribution with the normal tar -xvzf Mail-Cclient-x.xx.tar.gz and cd into the directory. You will then, unfortunately, have to edit the Makefile.PL by hand.  Specifically you will have to change INC argument to WriteMakefile() to be:

    "INC"    =>   "-I$CCLIENT_DIR -I/usr/include/imap",

This instructs the build process to look for the shared library in $CCLIENT_DIR and the headers in /usr/include/imap.

Then it is just a matter of calling:

perl Makefile.PL CCLIENT_DIR=/usr/lib
make
make install

Hope this helps someone else who gets bit by this annoyance.

UPDATE:
Turns out that I was grabbing an older version of Mail::Cclient ( version 1.1 specifically ) if you use Mail::Cclient 1.12 then the install process requires a few other RPMs:

• openssl-devel
• pam-devl e

And then you install it with:

   

perl Makefile.PL --cclient_dir=/usr/lib --with-pam --with-cclient-includes=/usr/include/imap/ --with-shared_cclient

Speaking at SCALE in Los Angeles

I nearly forgot to blog about this, but next Sunday (February 11th, 2007) I'll be giving a short introduction on using mod_perl 2.0 at the 5th Annual Southern California Linux Expo. I hope to see you there!

UPDATE: The slides for this talk can be found here.

And some people say programmers are boring

It is a common stereotype that computer programmers tend to be on the boring side.  Not Perl programmers it turns out.  We're a wacky bunch. Take for example our benevolent dictator Larry Wall, he's anything but boring.

Not to mention Schwern's Shirt.

And what other programming language has an elder like brian d foy who has violated the Posse Commitatus Act?

This is just a small sampling of Perl's fun and wacky side.  Personally, I think it's one of the reasons our community is so strong.

Introduction to Gantry, Bigtop, and Tentmaker screencasts

Thought I would mention that there are now two screencasts that show off some of the main features of the Gantry framework that I play a minor role in.

You can find the screencasts here under movies

Doing a LEFT OUTER join with DBIx::Class

I have recently been using DBIx::Class instead of the more popular Class::DBI. It has many advantages over Class::DBI that I won't go into here, but if you haven't used it yet you should definitely check it out.

One thing I found the other day is how to setup a special LEFT OUTER join query. If you have a situation where you need to do a LEFT OUTER join on your data, but only say in one particular script.  Or maybe a one off report that you won't be keeping around. You could go ahead and put in this relationship in your main model class, but for a one off that is a bit of overkill. 

What I hadn't thought about, was you can define those relationships from outside the MyModelClass.pm file itself.  Take for example a simple Artist -> CD relationship, where you want all artists even if they don't have any CDs: 

 

use ExampleSchema;

ExampleSchema::Artist->has_many('left_outer_albums' =>
                       'ExampleSchema::Cd', 'artist_id',
                       { join_type => 'LEFT_OUTER' } );

my $schema = ExampleSchema->connect('dbi:Pg:dbname=outer', '', '');

my $rs = $schema->resultset('Artist')->search(

    undef,

);

while( my $artist = $rs->next ) {
    print "Name: " . $artist->name . "\n";
    print "Albums: \n";

    foreach my $album ( $artist->left_outer_albums ) {
        print "\t" . $album->title . "\n";
    }
}

The nice thing about this is that this special left_outer_artists is defined and used in the one off and doesn't have to polute your main ExampleSchema::Artist relationships that might confuse someone. It may not be the best practice, but it is something to consider.

Learn something new about Perl every day

  Just when you think you know everything about Perl, something silly   rises up and shows you have ignorant you really are.

  How many times have you written code similar to this?

 
      my $filename = "/path/to/file.txt";
      my @dir_parts = split('/', $filename);

      my $file = pop( @dir_parts );
      my $path = join('/', @dir_parts );

or

      my ($name) = $filename =~ s/\/(.*?)$/o;

While I knew about the existance of File::Basename, the last time I looked at it I don't believe it was part of Perl Core.  I should have suspected, but now it is a standard Perl module that makes this trivial:

 
      use File::Basename;

      # Retrieve just the filename

      my $filename_only = basename($filename);

      # Get just the path in this filename

      my $path_only     = dirname($filename);

  You can get even fancier with the fileparse() function provided in
  this module.

      my ($base, $path, $suffix) = fileparse( $filename );

   Would yield the filename only in $base, the path in $path, and
   nothing in $suffix.  This is because we did not provide a regular
   expression to match on.

   If we instead used:

      my ($base, $path, $suffix) = fileparse( $filename, qr{\.txt} );

   And we ran it against $filename = '/home/frank/test.txt' and
   $filename2 = '/home/frank/test.doc' it would give us:

      Base: test
      Path: /home/frank
      Type: .txt

      and

      Base: test.doc
      Path: /home/frank
      Type:

   If the filename give to fileparse() does not match, it is not
   stripped from the basename.

It just goes to show that no matter how long you've been using Perl, or how much you think you know, there is always something out there you could be learning.

Been far too long...

It's been far too long since I've had time to post on this blog.  Time passes at a frightening pace doesn't it?  Here two tidbits of info you might be interested in:

 
1. Released a new version of   Apache::DB, Apache::DProf, and Apache::SmallProf. The new release fixes a bug where you couldn't use Apache::DProf under taint mode and allows you to specify anywhere on the file system for your dprof info to be dumped.  Previously it had to be relative to ServerRoot.
2. Gantry, yet another web framework, that I have been helping out with was finally released publically. I helped mostly with Gantry::Conf  a configuration abstraction interface.  It is essentially DBI for configuration files.  Right now it doesn't support everything I would like it to, but the hooks are all there and I plan on fleshing the rest out in the near future.

Apache debugging and performance tuning article

Looks like my article on Debugging and performance profiling mod_perl applications is up on www.perl.com.

Apache::DB, Apache::DProf, and Apache::SmallProf updates

Recently it came to my attention that there were some fairly serious bugs in some of the CPAN modules I maintain.  Stas Bekman was trying to use Apache::DProf to profile the performance of one of his projects and could not get it to work.

The problem essentially boiled down to the generally accepted best practice at how to determine if your code is running in mod_perl 1.x or 2.x doesn't work in all cases in all environments.  This is because of when the $mod_perl or $mod_perl2 modules are loaded in relation to the Apache lifecycle. 

Instead of using:

use constant MP2 => eval { require mod_perl; $mod_perl::VERSION > 1.99 };

We're going to use:

use constant MP2 => ( exists $ENV{MOD_PERL_API_VERSION} and $ENV{MOD_PERL_API_VERSION} >= 2 );

Because this environment variable is guaranteed to always be available and accurate.

I've uploaded a new distribution of these modules to CPAN just now and  it should make it to your favorite mirror in the next 24/48 hours. If you currently use these modules or the above technique to make your mod_perl code work in both an 1.x and 2.x environment you are encouraged to make these changes as well.

ModPerl::ParamBuilder released

I just released a new module to CPAN ModPerl::ParamBuilder which makes it much easier to build custom Apache directives for your mod_perl 2.0 applications.

You might be asking why you would want to do such a thing. The main reasons are:

   
• Custom directives are more efficient than using PerlSetEnv or PerlSetVar. They are evaluated only on server startup and not for each request like PerlSetVars
   
• It gives your application a more polished and professional        look and makes your configuration more intuitive for end users
   
• It's just plain cool

Assuming you're building an application called MyApp that will need to be passed various parameters such as database user, password, database name, database server address, etc. Here is how ModPerl::ParamBuilder fits in. 

First you create a separate module that will hold your custom directives. We'll call that MyApp::Params and would look like:

package MyApp::Params;
use ModPerl::ParamBuilder;
use base qw( ModPerl::ParamBuilder );

my $builder = ModPerl::ParamBuilder->new( __PACKAGE__ );

$builder->param('DBUser');

$builder->param('DBPass');

$builder->param('DBName');

$builder->param('DBServer');

$builder->on_off('AutoCommit');


$builder->load;


1;

Putting these directives to use in your Apache's httpd.conf is easy,
you just need to load your MyApp::Params module.

PerlLoadModule MyApp::Params


<Location /myapp>

  SetHandler perl-script

  DBUser apache

  DBPass secret

  DBServer 127.0.0.1

  DBName myapp

  AutoCommit On

  PerlResponseHandler MyApp::Main

</Location>

NOTE:You must use PerlLoadModule and not
the more common PerlModule Apache directive for your parameter module.  This is because Apache needs to load this module very early in the server startup so that it can read it's own configuration files.

To retrieve and use these directives from your application you add
the following to MyApp::Main:

use MyApp::Params;


my $params = MyApp::Params->new;

my $config = $params->get_config;


my $dbuser = $$config{'DBUser'};

# etc

Hopefully everyone will find this module as useful as I have. Personally, I think being able to build custom Apache directives easily is of the neatest features of mod_perl 2.0.

UPDATE: I've also put up a short short tutorial on how to use ModPerl::ParamBuilder.

Feel free to contact me if you have any questions or have a problem using it.

Sending E-mail from Perl

 

UPDATE: I've recently found an even better way of sending Email messages than using any of the information listed here.  Check out my new post on the subject for the details

I'm always amazed at how many people have trouble doing something as simple as sending E-mail from Perl or mod_perl.  I think this is because new Perl programmers are either unaware of CPAN or afraid to use it.  Trust me CPAN is your friend. :)  If you install the Net::SMTP module, which is part of the libnet CPAN package, on your system it is trivial to send plain text E-mail messages.

   

Here is a brief example:

   

# Create an instance of the module
  my $smtp = Net::SMTP->new( 'smtp.example.com') or die "Cannot connect to host: $!";
 

   

$smtp->to( 'recipient@example.com' );

  $smtp->data();
  $smtp->datasend("To: recipient\@example.com\n");
  $smtp->datasend("From: sender\@example.com\n");
  $smtp->datasend("Subject: Test Subject\n");
  $smtp->datasend("\n");
  $smtp->datasend("This is where the body of your message goes!\n");
  $smtp->quit();

   

This module assumes you know a little bit about the SMTP protocol, but there are tons of modules on CPAN that are even easier to use.  This one just happens to be the one I use the most.  Some others are:

   

 
• Mail::Sendmail
   
• Mail::SendEasy
   
• MIME::Lite
   
• Email::Sender::SMTP::Auth
   
• Email::Send
 

While I haven't personally used all of these modules, they all seem to have clean interfaces for sending E-mail from your Perl programs.  MIME::Lite especially is useful when you want to send attachments along with your message.  Hopefully this information helps you in your future projects!