IT Opinion: Automation and Automatons

Automation, Trust but Verify

Herman Buitenhuis over at the AMIS Technology blog had some adventures with backups scheduled using Oracle OEM Grid. Oracle Support fixed him up with a script to run and get things caught up, and life goes on.

But there's an important lesson in the incident. We come to depend on automation too much at times. Yes, the combination of Grid and RMAN are really handy and save lots of time. But having automated systems that are neater than sliced bread does not mean that we can totally depend on them. Things break, and that's when the DBA needs to know what is supposed to be happening, what is not happening and, to a greater or lesser degree, how to fix them.

The DBA should understand what the RDBMS and related systems are doing. If, for instance, you set up an instance with four copies of the control files, all on the same mount point, you need to get some more fundamental training on how Oracle works.

Tom Kyte (and many other members of the Oak Table Network), read big chunks of the documentation when a new version of Oracle comes out (particularly the Concepts manual). That's because it's only when you have the core understanding of the technology that you can move on to the spiffy features and advanced topics.

A database is a field of study, and like any field of study, you can skip the fundamentals and just dive in, but you'll be left with dangerous weak points in your knowledge. There is some reason that courses in technical subjects cover a lot of boring ground work before getting to the 'building laser-toting robots covered with blinkenlights' course. If you don't have the basics down, at some point your robots are going to either not work, or work too well, resulting in years of litigation and prison food.

The DBA should able to fall back on at least one technology to use for every automated system in place. There are a lot of DBA's working today who have had their entire experience in GUIs, with only a glancing and uncomfortable acquaintance with the command line. They sometimes discover they really need the command line in the midst of a crisis when, for instance, they have to access a system remotely or the GUI for the UNIX box they are working on suddenly decides to die. You need to know how to get all your vital functions accomplished at the command line. I'm not saying everyone has to remember all the syntax or use the command line in preference to GUI. But it's a simpler and more dependable way to do things, and the lack of complexity means it has less ways to fail than a GUI-based tool.

As to memorizing syntax, which I could never do worth squat, I've always loved an incident attributed to Albert Einstein. Someone asked him his telephone number. He said he didn't know it. The questioner was shocked, of course. But Einstein told him, "I have a lot of things to keep in my head. You can find my number in the telephone book. That's what it's for."

The Internet is our telephone book. Even if you've forgotten how to mount disks on a balky system that requires some preliminary settings before obliging, the odds are extremely good that you'll be able to find the error shown and its solution in 10 minutes on Google rather than tracking down a person who knows how to do the task and speaking with them (or worse yet, flying them in) or keeping all the data in your head, but neglecting some other important matters. (Or you could just be one of those people with photographic memories for problems and syntax and just remember everything. But I hate them, so I'm not going to discuss it further).

In short, when using automated systems, look at their actual results from time to time, and don't just accept their 'Success' messages at face value. Trust, but verify.