Over-engineering

One of the lessons I've learned all too well over the years is the cost of over-engineering. It's an easy trap to fall into, because once we start to get into a particular problem, and start breaking it down into smaller tasks, it's easy to see other problems that could also be solved using the same tools. Complication ensues, because we try to write code that covers every possible use case. This violates the agile programming rule of YAGNI (you ain't gonna need it), which is a good rule for avoiding over-engineering. But there's an even broader principle that also applies here, and that's the principle of doing more by doing less.


Let's say, for example, we need to send an email alert as part of our code. Our first cut does exactly that: it loads in an email library, assembles the message, and mails it to the recipient. But then we think, we don't want to have to push a new code base every time the recipient address changes, so we factor out the email address into a configuration setting. Then we realize we might want to change the text of the email, so we make that an external config file as well.

So far so good. But then we want to embed runtime data values into the email, so we add a templating system. And then we make the templating system configurable. And then we realize our templating system could have a lot of other applications as well, so we parameterize out all the possible variations for configuring a templating system. And so on.

Remember, all we needed to do was send an email. We can still do that, of course, but now it takes four configuration files and 19 parameters that have to be tuned the right way, AND we have to coordinate between systems because the invoice generation subsystem is using the same templating engine, so changing the email settings could screw up the invoices, and vice versa.

The root problem is that we had a tool, and saw what it could do, and tried to make it do more. And the sad part is that we succeeded. It does do more, which is precisely why it is so complicated to use.

What I like about functional programming is that it tends to head the opposite direction. Instead of becoming more powerful by doing more, it adds power by doing less. Instead of building an email class that can be configured to use sendmail/postfix/etc and parse messages, and derive MIME types and so on and so on, you just write a function that does one thing, and does it fairly stupidly. Maybe it takes an array of headers, and a body, and returns "true" if the required TO/FROM/SUBJECT headers are all present. Or it assembles the headers, body, and attachments into a correctly-formatted mail message as a raw string. Or it just pipes the raw string to a system process.

The thing is, each function is highly ignorant regarding the larger task to be accomplished. The function that checks all the headers doesn't necessarily know it's being used to validate email messages. All it knows is it's checking a list of strings to see if the list contains strings that start with "To:", "From:" and "Subject:". And to make this function more powerful, we make it dumber. We take away everything it knows about what headers are required for a valid email message. Change it so that it takes a list of strings to look for, and a list of strings to search. Now we've got a function that knows even less about the problem we're trying to solve, and is therefore useful for more problems. We can pass in any list of required strings, and any list of strings, and get back a true or false that tells us whether the required strings are all present.

This is the essence of the principle of separation of concerns. We want code that knows what it is doing, and neither knows nor cares why it is doing it. The tools map, filter, and reduce are powerful and ubiquitous in functional programming because they know exactly what they're doing without knowing or caring why we want them to do it. They do one thing, and only one thing, and that lets us use them to do just about everything. The less they do, the more powerful they become.