I was happily working on my project when one of the System Administrators poked his head into my office notifying me that the development DB server had to be rebooted.

"Why are you rebooting it?" I ask.

"We need to add more disk space."

"You need to reboot a Sun Fire Server to add disk space? Doesn't it have hot swappable drives?"

"Yes, but we need to reboot it."

"Okay."

They shut it down cold. No 'shutdown -h now'; just a hard power down. They bring it back up, my DB is corrupted. Ergh!

Walk over to the DBA. "Can you restore my DB from backup for me?"

"I was not backing up the development DB's," sayeth the DBA.

Burning... fire... anger... hatred... murderous intent...

Apparently, we will not be hitting the June 2nd release date, now. I have all of the SQL scripts that handle table creation, foreign key definitions, procedures, functions, etc. in CVS. But, all the data in the look-up tables is gone. And, that, actually, constituted quite a bit of data. Think of all the lookup tables you need for a University -- normalized department listing, majors, schools, classrooms, buildings, classes, etc.

So, sure I have code like this stored in CVS:

create table LUCOLLEGE
(
IDCOLLEGE NUMBER not null,
COLLEGEABBREVIATION VARCHAR2(4) not null,
COLLEGENAME VARCHAR2(255)
)
tablespace LDAPTEST_DATA
pctfree 10
pctused 40
initrans 1
maxtrans 255
storage
(
initial 16K
next 8K
minextents 1
maxextents 500
pctincrease 0
);
-- Create/Recreate primary, unique and foreign key constraints
alter table LUCOLLEGE
add constraint PK_LUCOLLEGE primary key (IDCOLLEGE)
using index
tablespace LDAPTEST_DATA
pctfree 10
initrans 2
maxtrans 255
storage
(
initial 16K
next 8K
minextents 1
maxextents 500
pctincrease 0
);
alter table LUCOLLEGE
add constraint UQ_LUCOLLEGE unique (COLLEGEABBREVIATION)
using index
tablespace LDAPTEST_DATA
pctfree 10
initrans 2
maxtrans 255
storage
(
initial 16K
next 8K
minextents 1
maxextents 500
pctincrease 0
);

So, it will be easy to recreate the structure. But, there is thousands of data rows that populated those tables. And, it was month's of detective work pouring over 97 hundred different main frames and department-run MS Access tables to track down the different data that was normalized. Bah!

Luckily, I do have 97% of the data written in flat-ASCII files because that is how the data was extracted for me during meetings that went something like:

"Yes, I have a partial listing of all of the different majors we offer here."

"Can I have access to it?"

"No, it's in a 7 year old Access file that is so horribly corrupted it cannot be copied from file system to file system."

"Uhhh... can you just dump me all the data in it to a flat file so I can Perl over it, fix it, and insert into the master Oracle tables."

"Maybe."

"Do it, or I publish the entirety of your email on the Internet including the one's from your 17-year-old mistress."

"Okay."

Let that be a lesson. Never trust a DBA to be doing back-ups. Never trust a Sys. Admin. from having a lose-power-button-hitting-finger. And, threats of public humiliation will get you everywhere.

J$
#!/usr/bin/perl
J=>money
;$_=ord$"<<s>>$J>,s-.-
$&*$'+$&-e&&y[%_(8)]]J]
&&print chop;print chr

Trackbacks

Comments

Yeah, I love that about pretty much all OSs. Adding more harddrive space on the fly is not exactly the easist thing in the world. If you just want to add more drives (/mnt/morespace) instead of making /data larger, then it is no big deal. Otherwise, you need one of those super fancy Veritas filesystems that allow on the fly resizing (others might make them as well...) Sucks to be j$.

Posted by gnubbs on May 29, 2003 06:35 PM

It just so happens that we do use vxfs on the production DB.

Posted by J$ on May 29, 2003 07:10 PM

I'm glad to hear that your competency level is still the exception and not the rule in the world of Academia.

Posted by alw9 on May 29, 2003 07:16 PM

After calming myself, I went to talk to the Sys. Admin.'s again; and things were not *exactly* as I had first thought.

Apparently, they had tried to gracefully shut the server down; but the Veritas software began hanging. Then, the Veritas stuff began freaking out because it stumbled upon some bad FS Voodoo. After a call to Veritas, the peeps there told them to do the cold shutdown, which they dutifully performed. When they brought the system back up, every single DB was fine... all but mine.

So, tonight, they are running a crazy uber-journal-Veritas-vxfs version of fsck to try and fix my DB. If that does not work, they are restoring it from tape. So, my precious DB is still there; and the Sys. Admin.'s are not as bad as my initial knee-jerk sputterings made them out to be.

But, I can still berate the DBA for not backing up the development server's DB's.

Posted by J$ on May 29, 2003 07:34 PM