IT Opinion: Tangents Starting from In-Memory DBs

IT Opinion: Memory is Fast. What 'Ya Gonna Do About It, and a Few General Performance Thoughts

Another round in the battle between those who look at the whole system (including the existing code) and come up with a performance solution vs. the 'gosh, memory is just so, so fast, let's put the whole DB in memory' school over at Kevin Closson's blog here. Dr Stonebraker, a respected figure in database theory, seems to be on a commercial tangent, promoting a specific approach to the RDBMS, as Kevin notes, somewhat sarcastically (ok, yes, completely sarcastically) in his posting.

Focusing on the real problem when scaling up or dealing with a performance problem is often difficult. It is tempting to fix the parts that are easy to fix and/or are in your control, rather then looking at the whole picture. It is far more difficult to plan logically rather than to fix reactively. Profit is the key, ultimately. If your technology investments don't make more money for you than they cost, then you're better off going back to writing things down on index cards and mailing them to clients. So it is our job to convince others in our organization to 'do the right thing' and work on enterprise-wide solutions rather than constantly run around corking leaks and duct-taping blowouts.

Performance problems tend to be the place where the great political divides show up in a company. Two common points of misunderstanding are storage and CPU. There is a common misconception that throwing faster CPU at a problem will always help. If the problem is in fact the speed, quantity, or arrangement of your storage, throwing CPU at the problem will add to your I/O delays by getting requests for I/O processed faster and queuing up at the already overtaxed storage array.

Things often get dicey when it is the DBA calling for the changes (since the database is inevitably pointed to as the culprit in any performance problem, even if it's the performance of the CFO's car). The DBAs, even the DBA group, often lacks the authority to influence major decisions on storage and network systems, and are often outvoted when deciding on the operating system, one of the most fundamental factors for a new system. The database is a large factor in many decisions, but the mixture of in-house networking and storage needs, religious affiliation for certain operating systems, etc. all combine. Sometimes this combination can place convenience and perception ahead of profitability, and that's dangerous.

Now throw into the mix younger members of the team who consider the term 'Open Source' to be a Good Computer Housekeeping Seal of Approval. This approach is generally based on the idea of coding everything (including security) into the application and using an open source 'data store', such as MySQL to hold the application's coat for it while it is out there conquering the world. The approach doesn't make use of any of the features specific to the RDBMS, preaching the ideology of being 'database neutral', and thus able to be converted to any RDBMS available with great ease. This just means that you can save money once when you change RDBMS, but every single minute you run your system wastes huge amounts of money as your team recodes all those RDBMS features you deliberately don't use. You could call this the 'save once, waste daily' school of RDBMS philosophy.

Add to all this several departments with little technical knowledge but a lot at stake (HR and budgeting for instance), and you have the whole confusing picture.

People who read this blog tend to be the people figuring out how to actually get the best value for their company's dollar when making software decisions and doing their best to make sure the right hardware decisions get made. So in all I've mentioned above I'm basically preaching to the choir, summarizing a few things that everyone deals with in performance optimization, and indeed enterprise computing as a whole, every day.

I guess my ultimate message would be that contention for resources leads to conflict, conflict can lead to litigation (or at least internal corporate warfare), and ultimately it just wastes money. Next time you are looking at a problem that damages your section of the technology, why not try to look at the motivation of everyone at the table and come up with a compromise that produces maximum profit for the company (rather than, say, a really cool new piece of equipment with lots of blinkenlights.)