Today’s unplanned System Outage
Sent 1/25/2007
———————-
For some reason (we are not quite sure yet why) the main mail server went down at 12:56pm. It was back on-line at 10:50pm
Fortunately “most” of the disaster recovery steps worked as planned and we were able to bring the server back on-line without any loss of mail delivered prior to the actual outage. Since “all” the disaster recovery steps did not work as planned it took us a bit longer then anticipated to bring the server back on-line. Therefore, some mail intended for delivery to you may have been returned to the sender as undeliverable if the retry settings on the sending server was surpassed.
This weekend we will be analyzing the outage to determine if it was a hardware failure or if it was the result of something malicious. We are tending right now to believe it was a hardware failure as that is what the indications are trending. As much as we might like to think it was some SPAMMER mad at us for blocking their mail.
We are also making plans and taking steps for additional hardware and software to be able to bring the servers back on-line within a much shorter time frame if a failure were to occur again in the future. By the way, this was the first full server crash we have ever had on one of the mail servers going all the way back to 1996! A pretty good longevity record that we hope to again reestablish.