![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
(no subject)
Ugh. Just as I was watching Babylon 5, I got a phone call saying that a library could reach our office but not the Internet. I got the same call from another library. Oddly enough, they could ping Yahoo.
I suspected a BIND crash, especially since vaxer.net's BIND crashed yesterday; it seemed likely that DNS info for www.yahoo.com would have gotten cached somewhere.
I went into the office and found that I could surf the Web just fine, but I couldn't ping the libraries' routers. Traceroute stopped at the firewall. Yup, sure enough, the firewall had locked up. One quick reboot and that problem was solved.
A few minutes later, as I was checking my mail for Nagios messages to explain the problem, I figured it out. Two people subscribed to a 500-person mailing list had turned on their vacation programs at roughly the same time, and they caused an undetectable loop that flooded the list with over 100 messages (throttled, I suspect, only by the speed at which our mail server was able to churn out the autoreplies that triggered more messages). I'm guessing about 60,000 messages went out in about 3 hours; that's not a lot in average (about 5 per second), but it was surely bursty.
So I unsubscribed the culprits, since I didn't have the GPS information needed to send a 20-kiloton rebuke, and everything seems to be back to normal.
Good thing I finished the new Harry Potter last night. I'd have been most put out if I'd had to put that book down -- mainly because it takes a lot of strength to lift it again.
I suspected a BIND crash, especially since vaxer.net's BIND crashed yesterday; it seemed likely that DNS info for www.yahoo.com would have gotten cached somewhere.
I went into the office and found that I could surf the Web just fine, but I couldn't ping the libraries' routers. Traceroute stopped at the firewall. Yup, sure enough, the firewall had locked up. One quick reboot and that problem was solved.
A few minutes later, as I was checking my mail for Nagios messages to explain the problem, I figured it out. Two people subscribed to a 500-person mailing list had turned on their vacation programs at roughly the same time, and they caused an undetectable loop that flooded the list with over 100 messages (throttled, I suspect, only by the speed at which our mail server was able to churn out the autoreplies that triggered more messages). I'm guessing about 60,000 messages went out in about 3 hours; that's not a lot in average (about 5 per second), but it was surely bursty.
So I unsubscribed the culprits, since I didn't have the GPS information needed to send a 20-kiloton rebuke, and everything seems to be back to normal.
Good thing I finished the new Harry Potter last night. I'd have been most put out if I'd had to put that book down -- mainly because it takes a lot of strength to lift it again.