At around 1am on the 13th we installed some pending Windows updates to the server and abound restarting it to activate the updates the operating system of the server failed.

After spending considerable time trying to resurrect the OS and bring the server back on line we were forced to replace the drive to save the data that was on the drive in the event that a system restore from backup proved ineffective.

Whatever was causing the issue (corruption or a compromise) was included in the backup because bringing the system on line after the restore we saw the same problem.

We ended up having to rebuild the server from scratch and then copying the data from the saved original drive and modifying the configuration of the software to access the saved data.

The server started accepting mail again around 7pm.  The web server was brought back on line around 8pm and with most of the additional services back to function around 9pm.

Within about 45 minutes of the mail server coming on line we could see that there were almost 800 inbound emails sitting in the queue waiting to be processed.

Incoming Spam Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

SpamAssassin Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Viruses Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

ClamAV Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Greylisting Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Server Status Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

Message Traffic Trend [summitinternetservices.com]
Date Range: 3/13/2009 to 3/13/2009

This report generated by SmarterMail 5.5.3348.Copyright© 2003-2009 SmarterTools Inc. All Rights Reserved
Sat, 14 Mar 2009 07:05:51 GMT

Posted via email from Summit Internet Services

When a sending mail server gets a retry order (in this case a 451 grey-listing response) they should requeue and resend their mail at a reasonable time frame.

The published Internet Standard for retrying to send mail is 30 minutes after the first delivery failure.

http://www.rfc.net/rfc2821.html

“…The sender MUST delay retrying a particular destination after one attempt has failed.  In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery…Experience suggests that failures are typically transient (the target system or its connection has crashed), favoring a policy of two connection attempts in the first hour the message is in the queue, and then backing off to one every two or three hours.”

Sometimes the sender is using a Microsoft Exchange Server which has a documented problem with this standard.  Microsoft documents the issue and provides a Registry fix

http://technet.microsoft.com/en-us/library/aa998772.aspx

We discovered that two of the SPAM Databases we have been using (although set to a low trust level) have become rather un-trustworthy. In isolation this would not present an issue but if taken in tandem could “could” result in a particular piece of email being falsely flagged as SPAM which if the score was high enough would then result in that email being blocked from delivery.

Those Databases have been removed from from our battery of tests.

Also we have stopped the server from adding the prefixes SPAM-Low and SPAM-Med from the subject line of suspect emails. The filters are doing a good enough job in identifying and stopping real SPAM that generally those flagged with the pre-fix might look “SPAMMY” but weren’t.

Checking the current stats:

Delivered
	Last 5 min.	Last hour	Last 24 hours
Local	32	239	4404
Remote	4	75	2214
Total	36	314	6618

---

Incoming Spam
	Last 5 min.	Last hour	Last 24 hours
Spam-Low	2	15	287
Spam-Med	1	10	138
Spam-High	9	60	1318
Blocked	23	313	9310

---

Greylisting
	Last 5 min.	Last hour	Last 24 hours
Blocked	18	265	5557
Allowed	10	196	3231

So total stopped SPAM in the past 24 hours has been 16,185 pieces of mail with 4,404 pieces delivered to your in-boxes.

Sent 4/11/2007

———————

We had a confluence of events that resulted in a small outage this afternoon.

We noticed that speed and performance on the mail server was lagging. Upon investigation we discovered that the performance of our connection to Qwest was not running at it’s full capacity. In fact it was running at a very slow DSL “type” of speed.

Performance was being further impacted on our degraded lines by one of our clients sending multiple emails with nearly 20MB attachments. On a normal day this would not have been an issue but due to the degraded state of the line the inbound and outbound transmission of these emails were severely impacting the available bandwidth for the rest of the mail. Essentially being a cork in the bottle.

Qwest was notified and while they were working and investigating the line (causing it to be taken down during their testing) we took the opportunity to install a new upgrade to our mail server software.

The upgrade contains quite a few feature updates that we had personally requested and are very happy to see
included. Additionally it sounds like a bunch of performance enhancements have been implemented that will make the server work even faster. I know that it appears that the web interface is faster although they do not mention this as being a feature

http://www.smartertools.com/Company/News.aspx?NewsItemID=204

Sent 1/25/2007

———————-

For some reason (we are not quite sure yet why) the main mail server went down at 12:56pm. It was back on-line at 10:50pm

Fortunately “most” of the disaster recovery steps worked as planned and we were able to bring the server back on-line without any loss of mail delivered prior to the actual outage. Since “all” the disaster recovery steps did not work as planned it took us a bit longer then anticipated to bring the server back on-line. Therefore, some mail intended for delivery to you may have been returned to the sender as undeliverable if the retry settings on the sending server was surpassed.

This weekend we will be analyzing the outage to determine if it was a hardware failure or if it was the result of something malicious. We are tending right now to believe it was a hardware failure as that is what the indications are trending. As much as we might like to think it was some SPAMMER mad at us for blocking their mail.

We are also making plans and taking steps for additional hardware and software to be able to bring the servers back on-line within a much shorter time frame if a failure were to occur again in the future. By the way, this was the first full server crash we have ever had on one of the mail servers going all the way back to 1996! A pretty good longevity record that we hope to again reestablish.

Sent 1/14/2007

—————————-

Well the upgrade on the mail server was completed last night with minimal difficulties and the server brought back on-line at around 7:30pm.

The decrease in delivered SPAM is almost spooky!

Granted it is a Holiday weekend so traffic will be lighter. However, looking at the log files as of about 2:30pm this afternoon from midnight last night the server is blocking 95.87% of all inbound mail as SPAM.

I know that the amount of SPAM that has made it through to my own in-boxes totaled just three from midnight to now. All three of those did have Opt-out links that I used.

Last Sunday volume on the server was 6,237 blocked as SPAM, 2,290 deleted as SPAM (total stopped = 8,527), 1,329 deliver as “clean” and 669 delivered but flagged as possibly SPAM (total delivered = 1,998) which represents 81.02%. As of right now the volume is 5,807 blocked, 455 deleted (total stopped = 6,262), 156 delivered “clean” and 114 delivered flagged (total = 270).

It will certainly be interesting to see how we fair when the workweek restarts on Tuesday. If you have ANY issues or CONCERNS please let us know at either: dave@microworks.net or by calling at 480-610-8234.

Sent 1/13/2007

——————–

IMPORTANT INFO – PLEASE READ THE ENTIRE MESSAGE

We are planning on upgrading the Mail servers this weekend (1/13-1/15). Hopefully you will see a dramatic decrease in the amount of SPAM that gets through to your in-boxes.

Currently 80% of the mail that is directed at our servers is classified as SPAM and deleted but a considerable amount still gets through (I had 52 SPAM emails make it through yesterday to the various accounts that I use and monitor).

The biggest single help to our Anti-SPAM efforts will be new technology included with our software called “Greylisting”. Greylisting can mean different things so here is what it means on our server.

White-listing is when you have a list of “trusted” addresses that you allow through without testing. Black-listing is when you have a list of addresses that are known to be bad and are blocked BEFORE they are accepted for SPAM testing.

The Grey-Listing technology that we will be using basically tells the remote mail server sending email that we are having a “technical” issue and to retry later. A legitimate mail server puts the mail it is trying to deliver back into its queue and will attempt to redeliver at it’s preset time schedule. We have no control of that time frame. Some servers may try right away while others may wait a bit longer before retrying to send.

(more…)

Sent 12/7/2006

———————-

With recent reports in various media, including the New York Times, showing that 90 percent of all email is now SPAM I was wondering how our Anti-Spam efforts were holding up.  It has been a while since I sat down and checked our stats.

During a review that covered the last part of June and the first half of July 68.33% of the inbound email was prevented from being delivered for failing our anti-SPAM tests.  With a total of 218,728 pieces blocked while allowing 101,376  pieces of mail to be delivered (17,069 of that total were flagged as possible SPAM).

A review the following month saw the percentage of blocked inbound mail reach 78.97%.  228,136 pieces were blocked with 60,747 delivered (15,063 flagged as possible SPAM).

This morning I undertook to look back over the numbers for the past month.

The blocked total is now up to 86.74%!  That represents 347,817 pieces of mail that have either been blocked from delivery (259,232) or deleted automatically (88,585) by the server after the mail failed our weighted testing scheme.  53,165 pieces were delivered with 27,771 of those pieces identified with the “possible” SPAM flags of “SPAM-LOW” or “SPAM-MED”.

(more…)

Sent 7/31/2006

———————

Hopefully you will have noticed a significant decline in the amount of SPAM making it through to your mailboxes over the last month or so.

The first 30 days (2/28/06 – 3/28/06) of the new mail server had us stopping 159,814 messages from being delivered to your mailboxes out of a total of 266,549 that were sent which represented 59.96 percent. On weekends the rate was hovering around 80-85% stopped.

By the time we got to 6/20/06 – 7/18/06 period we were stopping 218,728 out of 320,104 or 68.33% with most of the improvement occurring during the weekday since the weekend rate was still in the +80% range.

The overlapping time frame that we are looking at right now of 7/4/06 – 8/1/06 shows that the rate is still climbing. If trends for the next two days continue to hold true we will see the mail servers having stopped 220,771 messages identified as SPAM while allowing a total 40,175 to be delivered for a percentage of 84.60% out of 260,946 pieces of total mail! Of the total the server allowed to be delivered 14,920 pieces of mail that it thought “might” be SPAM and tagged with the designation of either SPAM-LOW or SPAM-MED.

(more…)

Sent 3/10/2006

—————————-

Greetings,

We would like to give you an update on the progress of fine tuning the new mailserver.

To date (12:01am 3/10/06) the server has received 78,809 messages (avg of 7,880 per day) and sent 11,022 messages since going on-line on 2/28/06 – 10 days ago. The total size of these messages have been 1.9 Gigabytes received and 2.3 Gigabytes sent.

We are certainly making progress on reducing the amount of SPAM that is making through the server and to the mailboxes.

Date Blocked Deleted
2/28/06 0 891
3/1/06 0 779
3/2/06 0 1,159
3/3/06 0 1,509
3/4/06 958 1,607
3/5/06 1,640 1,592
3/6/06 2,161 1,719
3/7/06 2,487 1,587
3/8/06 2,659 1,445
3/9/06 2,755 1,387

Sub Total 12,660 13,675

(more…)