Computer hell

I created a mailing list of all the users on our system and sent them a message announcing the spam filter.

The server crashed.

Ahem: THE server crashed. Web, mail, SQL, etc.

I rebooted it. It crashed. I rebooted it. It crashed. It would crash within 30 seconds of booting -- not enough time to prevent the problem even if I knew what it was.

I booted into single mode. I figured out how to get read-and-write access to the hard drives (mount / -o remount,rw). I restored /etc from tape and rebooted.

Crash.

I booted back into single mode. I changed the server so that it wouldn't start the mail process when it booted. I rebooted.

Now that's funny... None of the initialization scripts were running. After a LONG time, with the help of #linuxhelp on irc.undernet.org, I realized what was going on.

(Geeky details: I had restored a GNU tar backup without the --preserve flags, so /etc/init.d/rc?.d contained not symlinks but 0-byte files.)

Once the system was running again (minus the mail stuff), our consultant could come in, sweep out the mail files that were causing the problem, and start the mail server.

All was well.

And then I sent another message to all the users.

WHAM.

So the good news at that point was that SpamAssassin had fuck-all to do with the problem (almost certainly), because it wasn't even being used. The bad news is that it's either something that's wrong with the mailing list or something wrong with the server.

And indeed there are some I/O errors in the logs -- so tonight I get to fsck the root filesystem. Late enough that nobody would mind, early enough that I might be able to donate this damn dining room set and get something accomplished at home. (Hard to do, with Cowboy Bebop, 8 Mile and Spirited Away on loan from the library. Too bad my chemically-assisted willpower fades just about the time I get off work.)

OH JOY!! I get to stay really late! What more could a guy ask?

(Well, at least most of Friday I'll be using unofficial comp time rather than burning personal leave.)

All in a night's work

Reiser fsck detected problems that can only be fixed by --rebuild-tree. It recommends a full backup before you try this, so that's what I'm doing now. No idea how long all of this will take.

I figured out that you have to do this from a rescue disk; I'm copying the latest stable reiserfsck to a floppy because the one SuSE shipped is "experimental". Sheesh.

Fought with Jodi on the phone -- I'm not sure why. She seems to be unhappy that I'm working late. I truly don't get that -- it's not like I'm standing her up. Shit happens, I get to fix it, and that's all there is. Why add strife to a situation that already sucks? I'll get home eventually, and I'm sure they'll understand if I'm a bit tired.

The latest stable reiserfsck is rebuilding the tree. I'm guessing it takes about 20 minutes (gotta love progress meters).

Finally done for the night

reiserfsck cleaned up the problem, I think. Time to let our consultant know and see what he can come up with tomorrow...