At around 1am on the 13th we installed some pending Windows updates to the server and abound restarting it to activate the updates the operating system of the server failed.

After spending considerable time trying to resurrect the OS and bring the server back on line we were forced to replace the drive to save the data that was on the drive in the event that a system restore from backup proved ineffective.

Whatever was causing the issue (corruption or a compromise) was included in the backup because bringing the system on line after the restore we saw the same problem.

We ended up having to rebuild the server from scratch and then copying the data from the saved original drive and modifying the configuration of the software to access the saved data.

The server started accepting mail again around 7pm.  The web server was brought back on line around 8pm and with most of the additional services back to function around 9pm.

Within about 45 minutes of the mail server coming on line we could see that there were almost 800 inbound emails sitting in the queue waiting to be processed.

Incoming Spam Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

SpamAssassin Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Viruses Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

ClamAV Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Greylisting Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Server Status Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Message Traffic Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

This report generated by SmarterMail 5.5.3348.Copyright© 2003-2009 SmarterTools Inc. All Rights Reserved
Sat, 14 Mar 2009 07:05:51 GMT

Posted via email from Summit Internet Services

When a sending mail server gets a retry order (in this case a 451 grey-listing response) they should requeue and resend their mail at a reasonable time frame.

The published Internet Standard for retrying to send mail is 30 minutes after the first delivery failure.

http://www.rfc.net/rfc2821.html

“…The sender MUST delay retrying a particular destination after one attempt has failed.  In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery…Experience suggests that failures are typically transient (the target system or its connection has crashed), favoring a policy of two connection attempts in the first hour the message is in the queue, and then backing off to one every two or three hours.”

Sometimes the sender is using a Microsoft Exchange Server which has a documented problem with this standard.  Microsoft documents the issue and provides a Registry fix

http://technet.microsoft.com/en-us/library/aa998772.aspx

We discovered that two of the SPAM Databases we have been using (although set to a low trust level) have become rather un-trustworthy. In isolation this would not present an issue but if taken in tandem could “could” result in a particular piece of email being falsely flagged as SPAM which if the score was high enough would then result in that email being blocked from delivery.

Those Databases have been removed from from our battery of tests.

Also we have stopped the server from adding the prefixes SPAM-Low and SPAM-Med from the subject line of suspect emails. The filters are doing a good enough job in identifying and stopping real SPAM that generally those flagged with the pre-fix might look “SPAMMY” but weren’t.

Checking the current stats:

Delivered
	Last 5 min.	Last hour	Last 24 hours
Local	32	239	4404
Remote	4	75	2214
Total	36	314	6618

---

Incoming Spam
	Last 5 min.	Last hour	Last 24 hours
Spam-Low	2	15	287
Spam-Med	1	10	138
Spam-High	9	60	1318
Blocked	23	313	9310

---

Greylisting
	Last 5 min.	Last hour	Last 24 hours
Blocked	18	265	5557
Allowed	10	196	3231

So total stopped SPAM in the past 24 hours has been 16,185 pieces of mail with 4,404 pieces delivered to your in-boxes.