Computer hell
May. 13th, 2003 03:43 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sylvarI created a mailing list of all the users on our system and sent them a message announcing the spam filter.
The server crashed.
Ahem: THE server crashed. Web, mail, SQL, etc.
I rebooted it. It crashed. I rebooted it. It crashed. It would crash within 30 seconds of booting -- not enough time to prevent the problem even if I knew what it was.
I booted into single mode. I figured out how to get read-and-write access to the hard drives (mount / -o remount,rw). I restored /etc from tape and rebooted.
Crash.
I booted back into single mode. I changed the server so that it wouldn't start the mail process when it booted. I rebooted.
Now that's funny... None of the initialization scripts were running. After a LONG time, with the help of #linuxhelp on irc.undernet.org, I realized what was going on.
Once the system was running again (minus the mail stuff), our consultant could come in, sweep out the mail files that were causing the problem, and start the mail server.
All was well.
And then I sent another message to all the users.
WHAM.
So the good news at that point was that SpamAssassin had fuck-all to do with the problem (almost certainly), because it wasn't even being used. The bad news is that it's either something that's wrong with the mailing list or something wrong with the server.
And indeed there are some I/O errors in the logs -- so tonight I get to fsck the root filesystem. Late enough that nobody would mind, early enough that I might be able to donate this damn dining room set and get something accomplished at home. (Hard to do, with Cowboy Bebop, 8 Mile and Spirited Away on loan from the library. Too bad my chemically-assisted willpower fades just about the time I get off work.)
OH JOY!! I get to stay really late! What more could a guy ask?
(Well, at least most of Friday I'll be using unofficial comp time rather than burning personal leave.)
The server crashed.
Ahem: THE server crashed. Web, mail, SQL, etc.
I rebooted it. It crashed. I rebooted it. It crashed. It would crash within 30 seconds of booting -- not enough time to prevent the problem even if I knew what it was.
I booted into single mode. I figured out how to get read-and-write access to the hard drives (mount / -o remount,rw). I restored /etc from tape and rebooted.
Crash.
I booted back into single mode. I changed the server so that it wouldn't start the mail process when it booted. I rebooted.
Now that's funny... None of the initialization scripts were running. After a LONG time, with the help of #linuxhelp on irc.undernet.org, I realized what was going on.
(Geeky details: I had restored a GNU tar backup without the --preserve flags, so /etc/init.d/rc?.d contained not symlinks but 0-byte files.)
Once the system was running again (minus the mail stuff), our consultant could come in, sweep out the mail files that were causing the problem, and start the mail server.
All was well.
And then I sent another message to all the users.
WHAM.
So the good news at that point was that SpamAssassin had fuck-all to do with the problem (almost certainly), because it wasn't even being used. The bad news is that it's either something that's wrong with the mailing list or something wrong with the server.
And indeed there are some I/O errors in the logs -- so tonight I get to fsck the root filesystem. Late enough that nobody would mind, early enough that I might be able to donate this damn dining room set and get something accomplished at home. (Hard to do, with Cowboy Bebop, 8 Mile and Spirited Away on loan from the library. Too bad my chemically-assisted willpower fades just about the time I get off work.)
OH JOY!! I get to stay really late! What more could a guy ask?
(Well, at least most of Friday I'll be using unofficial comp time rather than burning personal leave.)
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
Re: #linuxhelp on Undernet? HAHAHAHAHAHA
Date: 2003-05-13 03:02 pm (UTC)I wouldn't have thought to ask in #linux anyway -- I figured the question would be off-topic. Perhaps I'm a bit too polite.
Does tar cf --preserve /st0 / seem like a reasonable way to do the backup?