It's a mystery

It's funny how things work out despite everyone's best efforts otherwise.

Last June, a division in our company submits a request to upgrade one of their business-critical applications to the latest version. It gets added to the project list and requires quite a bit of coordination since it involves a new server, a new version of Oracle, and new storage.

Our Oracle DBA's indicate it will take about two weeks and schedule it for August. As you might expect, the schedule slips into Septembeer, but really the Oracle DBA's do a good job of not letting it slip any further.

The Oracle DBA tries the upgrade in a test environment and runs into several problems. With the help of the application's support people, he gets them resolved but requests that we push the install date back one week to let him attempt the install again from beginning to end.

The representative from the division goes ballistic. It's been *five months* since she requested this upgrade - why do we keep delaying? I frown when she says this in a meeting and count on my fingers... July, August, September. Seems like three months to me. The user keeps saying "five months" over and over, but I keep my mouth shut - no need to torpedo my career because the user can't add.

The Oracle DBAs sigh and cave in. So we start getting the hardware set up. The server is no problem - we have plenty and to spare. The disk is another issue. For reasons surpassing understanding, we don't control our disk farms - another office in Arizona does. We ask them to honor the scheduled request for disk. The SysAdmin says he'll "try to get to it". Our wailing appears to fall on indifferent ears.

Meanwhile I'm setting up our Linux server to meet corporate standards, which means I don't do a default installation - no need to enable a web server, for instance. When it's done I hand it off to our Oracle DBA, who begins the Oracle install.

Unfortunately, the install fails. Seems we're missing some libraries. The DBA calls Oracle support, then brings the phone over to my desk. Seems that Oracle can see that we don't have the default Linux installation - and shut down the support call right then and there. Because we didn't start with the default installation, Oracle won't help us figure out which libraries we're missing. The woman from Oracle actually says, "It's like you started with a Tudor house and want to turn that into a ranch house. You can't."

It is on the tip of my tongue to tell her about my log cabin built from the bodies of Oracle support engineers, but I refrain. Instead I tell her that I don't want analogies, I want a list of necessary libraries. She refuses to provide them - she said it could be thousands of libraries. She smugly states that she's told this same thing to a hundred guys like me, and they all end up reinstalling the OS to the default installation.

By now my dander is up, but I simply end the support call - arguing with the support engineer isn't going to get me anywhere and we're under quite a bit of time pressure, as there's another mission-critical project waiting in the wings for us to start. Instead, the Oracle DBA sends me the error message he gets during the install. I plug it into the interweb and come back with some pages about a needed library.

I install that library, find that I'm missing a few dependencies, and install those as well. The Oracle DBA attempts another install, and it fails with a different message. Again I search the web, and find that the error messaage indicates we're missing another library. I install it, the DBA attempts the install - and it works. It didn't take thousands of libraries, only about a dozen.

From then on the problems evaporate. The guy in Arizona seems to have actually provided the disk without telling anyone, so we mount it. The rest of the upgrade goes without a hitch and is actually completed ahead of schedule. The user is ecstatic and tells the DBA he's awesome.

Me, I consider this my victory over the Tudors. I used to do technical support, and I know the difference between someone genuinely trying to help you and someone who will throw up whatever obstacles they can in an attempt not to do any work.

So despite an impatient user, a recalcitrant storage admin, and an obtuse Oracle support engineer, the project was completed successfully. How was that possible? I don't know... it's a mystery.