One consistent theme I have seen in 25 years in software has been the regular reinvention of some common technological building blocks. Time and again, well-meaning developers and organizations spend hard-earned effort building things they really should buy / borrow / steal, paying the opportunity cost of not building more of their product or service. It's been alternately surprising and bemusing, and I'll admit I've fallen victim to the temptation more than once as well. In this post, I'd like to list a few common reinventions and suggest why they recur.

Common Reinventions

At almost every organization I've worked at, I've seen at least one of these reinvented:

• Logging system
• Message queue
• O/R mapping layer
• ETL
• Job scheduler / work queue

It's a safe bet that if your organization is larger than a certain size, or has been around longer than a bit, you may have (re)built one of these systems as well. Almost certainly, you can think of others as well.

Why Reinvent These Wheels?

So what is going on here? It's not like we don't have multiple solid open source and commercial implementations of all of these. I think there are a few common factors at work:

• Requirements seem simple. Almost every developer can outline in a straightforward way what one of these systems would do.
• Implementation seems simple. Nothing here seems like rocket science. A logging system just has to get log messages from online systems into offline storage. How difficult could that be?
• Each seems like a fun and interesting, yet tractable, challenge. Why should we use a "heavyweight" off-the-shelf system, when all we need (seemingly) is a small subset of what product X does. Let's just build it ourselves.

Deceptive Simplicity and the 80/20 Rule

The truth is that it's actually not that hard to get the basics going. But the simplicity is deceptive.

There is an 80/20 rule at play. While a competent developer can knock something out in a few hours or days that covers the first 80%, correctly solving the remaining 20% of the problem takes an order of magnitude more effort than it appeared at first blush. In a logging system or message queue, for example, you rapidly run up against fun distributed systems problems around node failures, network failures, duplication and ordering, etc. We need to make the storage fast, efficient, and reliable. Ditto the network transport. The devil is in the details.

Focus and Tradeoffs

But even if we believed we could build a simpler, proprietary logging system or ETL pipeline, should we? I'm all for building where it makes sense, but let's not lose sight of the tradeoff here -- in the form of opportunity cost. Fortunately these are all solved problems, and have been for a long time. And they are not typically our organization's core business. So they are, to borrow the Jeff Bezos phrase slightly out of context, "undifferentiated heavy lifting".

What would we be building if we did not invest effort into one of these? It might be fun, but I'll submit that we probably have better uses of our limited time and resources.

What do you think?