Testing Exchange Anti-spam Configuration

The Exchange anti-spam configuration needs to be tested to ensure that:

• spam filters are actually blocking messages as intended
• false-positives are minimized so that anti-spam is not causing legitimate messages to be blocked

This document provides some general (not exhaustive) guidance for testing your configuration.

Testing anti-spam filtering can be difficult because multiple types of filtering are usually applied to a message, and often it is difficult to replicate the scenarios which anti-spam filters are attempting to block.

• Having a good understanding of how each type of filter works is essential (refer to the documentation for each type of filter).
• A basic understanding of email message headers and SMTP may be required for some tests.
• When conducting tests, be aware that test messages sent from within the internal network are often set to bypass anti-spam filters. Therefore, some tests may need to be performed from an external network.
• In some cases, you may need to temporarily adjust your configuration to be able to test specific scenarios.
• Some settings or types of filters may be able to be tested at an individual mailbox level first, before they are adjusted for the entire organization (SCL Junk Threshold, Allowed Senders and Blocked Senders, for instance).

Testing Legitimate Messages Are Not Being Blocked

Testing that legitimate messages are not being blocked should check not only that messages are being delivered, but should also include inspection of message headers to ensure that SPF status, SCL and Sender Reputation scores are being assigned appropriately. Obviously, the mailbox test messages are sent to will need to have Anti-Enabled and Anti-Spam Bypass disabled.

Some message headers which are useful to check when performing these tests include:

• Received (to obtain the IP address of the sender)
• X-MS-Exchange-Organization-SenderIdResult
• Received-SPF (the IP address shown here can also be useful)
• X-MS-Exchange-Organization-Antispam-Report (often shows the reason for acceptance or blocking)
• X-MS-Exchange-Organization-SCL
• X-ReturnPathSenderScore (custom header added by QSS Exchange Anti-Spam Toolkit when Sender Score is enabled)

If none of the above headers are present on a message, then it is likely that the message bypassed anti-spam filters due to artificial testing conditions and the test was therefore invalid. Sometimes the X-MS-Exchange-Organization-Antispam-Report header will indicate that anti-spam filtering was bypassed.

When configuring Transport Rules to filter messages, configuring the Transport Rule to add a custom header may be useful to be able to more easily track which Transport Rule processed that message. This can also be determined from Exchange Message Tracking logs using PowerShell.

Testing Filters Are Blocking Spam

Testing that anti-spam filtering is actually blocking messages as intended will require specific tests for each type of filter and is in general more difficult. Using a third-party may assist in testing some of the scenarios. Ongoing monitoring of anti-spam filtering is recommended. Monitoring of anti-spam performance will always be necessary when a new environment is configured or if significant changes are made.

We strongly recommend using the Spamhaus Blocklist Tester service, which allows you to perform an automated test for some of the anti-spam filters which are more difficult to test. Signing up for the free Spamhuas Data Query Service provides access to more tools and is recommended.

• Filters which use static lists, such as allowed or blocked senders or recipients, or allowed or blocked words or phrases, can be tested by deliberately generating test messages which you expect the filters to block.
• Testing IP Allow or Block Lists will likely require modifying the configuration to add test entries for valid external IP addresses, as the connecting IP address cannot be easily spoofed (unlike sender email addresses and domains).
• Testing that an IP Allow or Block List Provider is blocking mail as intended is nearly impossible, unless the service provides a testing facility. Apart from being unable to easily spoof the connecting IP address, the fact that the IP address needs to actually be on the IP Allow or Block List can make finding an IP address to test difficult. Some services provide a testing facility for their own list. We recommend testing that legitimate mail is not being blocked and monitoring performance in production.
• Testing IP Allow and Block List Providers using the nslookup command (outside of Exchange) is straightforward, and is recommended to ensure that they can actually block mail in your environment. Refer to IP Allow List Providers & IP Block List Providers, which explains how these services work, and how to use the nslookup command to test your DNS Configuration.
• Testing URL Block List Providers is quite easy once you have obtained a domain name which is on the list. Sometimes the website of the URI DNSBL will show recently-added entries, which can then be used to construct a test message containing a URL which is known to be blocked. Testing URLs and domains from recently-received spam messages may allow you to find a domain which is listed and can be used for testing. URL Block List Providers can also be tested using nslookup in the same way as IP Allow and Block List Providers.
• While the Content Filter can be easily tested in terms of Allowed Phrases and Blocked Phrases, testing the more sophisticated message analysis features can only be done by inspecting SCL headers on real incoming messages.
• The Recipient Filter Recipient Validation feature can be tested by deliberately attempting to send a message to a non-existent recipient, using a Telnet session to the SMTP server.