Computer hell
May. 13th, 2003 03:43 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sylvarI created a mailing list of all the users on our system and sent them a message announcing the spam filter.
The server crashed.
Ahem: THE server crashed. Web, mail, SQL, etc.
I rebooted it. It crashed. I rebooted it. It crashed. It would crash within 30 seconds of booting -- not enough time to prevent the problem even if I knew what it was.
I booted into single mode. I figured out how to get read-and-write access to the hard drives (mount / -o remount,rw). I restored /etc from tape and rebooted.
Crash.
I booted back into single mode. I changed the server so that it wouldn't start the mail process when it booted. I rebooted.
Now that's funny... None of the initialization scripts were running. After a LONG time, with the help of #linuxhelp on irc.undernet.org, I realized what was going on.
Once the system was running again (minus the mail stuff), our consultant could come in, sweep out the mail files that were causing the problem, and start the mail server.
All was well.
And then I sent another message to all the users.
WHAM.
So the good news at that point was that SpamAssassin had fuck-all to do with the problem (almost certainly), because it wasn't even being used. The bad news is that it's either something that's wrong with the mailing list or something wrong with the server.
And indeed there are some I/O errors in the logs -- so tonight I get to fsck the root filesystem. Late enough that nobody would mind, early enough that I might be able to donate this damn dining room set and get something accomplished at home. (Hard to do, with Cowboy Bebop, 8 Mile and Spirited Away on loan from the library. Too bad my chemically-assisted willpower fades just about the time I get off work.)
OH JOY!! I get to stay really late! What more could a guy ask?
(Well, at least most of Friday I'll be using unofficial comp time rather than burning personal leave.)
The server crashed.
Ahem: THE server crashed. Web, mail, SQL, etc.
I rebooted it. It crashed. I rebooted it. It crashed. It would crash within 30 seconds of booting -- not enough time to prevent the problem even if I knew what it was.
I booted into single mode. I figured out how to get read-and-write access to the hard drives (mount / -o remount,rw). I restored /etc from tape and rebooted.
Crash.
I booted back into single mode. I changed the server so that it wouldn't start the mail process when it booted. I rebooted.
Now that's funny... None of the initialization scripts were running. After a LONG time, with the help of #linuxhelp on irc.undernet.org, I realized what was going on.
(Geeky details: I had restored a GNU tar backup without the --preserve flags, so /etc/init.d/rc?.d contained not symlinks but 0-byte files.)
Once the system was running again (minus the mail stuff), our consultant could come in, sweep out the mail files that were causing the problem, and start the mail server.
All was well.
And then I sent another message to all the users.
WHAM.
So the good news at that point was that SpamAssassin had fuck-all to do with the problem (almost certainly), because it wasn't even being used. The bad news is that it's either something that's wrong with the mailing list or something wrong with the server.
And indeed there are some I/O errors in the logs -- so tonight I get to fsck the root filesystem. Late enough that nobody would mind, early enough that I might be able to donate this damn dining room set and get something accomplished at home. (Hard to do, with Cowboy Bebop, 8 Mile and Spirited Away on loan from the library. Too bad my chemically-assisted willpower fades just about the time I get off work.)
OH JOY!! I get to stay really late! What more could a guy ask?
(Well, at least most of Friday I'll be using unofficial comp time rather than burning personal leave.)
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
#linuxhelp on Undernet? HAHAHAHAHAHA
Date: 2003-05-13 01:29 pm (UTC)I know this group, only too well. Most of them are either disgruntled people who have gotten tired of the complete assholes in #linux or total newbs. In #linux we tend to kick people over to #linuxhelp when they have a question that we know is in the FAQ or can be found via google, particularly if it can be found with the I-Feel-Lucky button.
Actually, the fact that you were able to get help and that it worked for you is pretty cool, as that channel has been notorious for not being able to help very well.
Just so you know, there is a reason why 98% of the ops in #linux are NOT in #linuxhelp. The #linux ops don't have the patience for people asking questions that are definately in the manuals somewhere, or questions that can be found simply by reading a little bit.
With your problem, I can see why you would go looking for help, and I am glad that #linuxhelp was able to help you out. It is something that would have confounded me for a while too.
It does interest me concerning the reasoning behind you needing to restore /etc from backup. What led you to do this in the first place?
Re: #linuxhelp on Undernet? HAHAHAHAHAHA
Date: 2003-05-13 03:02 pm (UTC)I wouldn't have thought to ask in #linux anyway -- I figured the question would be off-topic. Perhaps I'm a bit too polite.
Does tar cf --preserve /st0 / seem like a reasonable way to do the backup?