Technical Recommendations Regarding Mailing Lists
“Even if you get any letter, you won’t be able to read it”
(Mark Twain)
We have already explained how to create mailing lists, how to improve them and make them more effective. We presented metrics which help to build the sender’s reputation. We told you about the Postmaster Mail.Ru interface which allows you to track them. Many companies, both those in the first stages of development, and very successful ones, forget these rules, that causes them trouble with email delivery, problems with technical support, etc. But we hope that you are not among them.
So, your project is gaining popularity, users like it and you plan to stay in touch with them. You are familiar with administrative requirements (which we explained earlier in Russian) and are going to organize a mailing list responsibly and with no spam for the users ready to receive your email blast. Or maybe you just want to set up a corporate email system. You set up the mail server from the distribution kit, write a script, launch and… 70% of emails do not reach their destination, 15% of them will be sent to the Spam folder, and the rest of the recipients can’t read the letter. This article looks in detail at what should you do to keep this from happening.
Why do “right-from-the-box” solutions not always work and other recommendations are needed? Email is one of the oldest Internet protocols, which has survived to this day with few changes, but it is accompanied by a lot of corresponding standards and provides no documented best practices. Email standards themselves have no methods for authenticating a user or for preventing unwanted mass mailings or spam. This is why all email servers execute a range of additional verifications in order to filter out unwanted messages (which can exceed 90%).
We have gathered some useful information which will help you to pass all of these additional verifications. Many of these recommendations are optional, but the more you implement the fewer problems you will encounter with email deliver.
Recommendations For Email Server Administrator:
Let’s start with… the IP-address. The provider, hoster or registrar will provide you with an IP-address (or a network). What should you do in order to use the selected address for email server? Is it suitable?
1. Make sure this is a real and static address.
Most email servers restrict receipt from dynamic addresses and from networks with address translation. The reason for this is that such networks host end users’ computers, which do not deliver email. This means that all emails from those networks are considered spam sent by bots. This is why such networks have become notorious.
2. Check the whois information using the whois utility or online services (they can be easily found through a search). Here is what you should pay attention to:
2.1 The network should have a correct description. If the description contains information indicating that the network is dynamic or designed for sending data to individual end users (e.g. ADSL), you will likely encounter many problems while using it.
2.2 The network should provide abuse-contacts for reports, and those contacts should be real and responsive. Otherwise the network (and your IP) might be put on blacklists in the event of spam-incidents from any of its hosts.
2.3 If the network is European (RIPE registrar), the network status should be ASSIGNED, ASSIGNED PA or ASSIGNED PI, meaning that this network is used. Many registrars forget to mark that the network is in use, and some clients restrict receipt of email from the unused networks.
Ask the registrar to enter the correct information in the network description, but also remember that many potential recipients “remember” old information. Because of this, you should initially use an IP-address from properly administrated networks with a good reputation.
3. Check and set up the PTR-record for the IP-address. The PTR-record should specify the real host name. It is recommended that
3.1 this name should be different from the dynamic host name — it should not contain such words as “dynamic”, “adsl”, “nat”, or long sequences of characters or number groups. For instance, server.example.com is a good, suitable PTR-record.
3.2 If you are planning to send large amounts of outgoing emails from several servers, it is good to assign names to the servers according to the standard “DNS Naming Convention for Outbound Internet Email Servers” and create a zone as required by this document, and a mxout A-entry.
3.3 A PTR-record’s name should resolve back to the same IP-address, i.e. A- or CNAME-entry server.example.com should exist and resolve to the IP-address of the example server. If the server will support several domains, there is no need to create PTR-records in every domain — a single entry in any of the domains will suffice. During DNS review you should use a third-party DNS-server in order to make sure that the entries are visible from the outside.
PTR-records are managed by the IP-network service provider, for example, by the internet or hosting provider. This can usually be done via the hosting control panel or through the provider’s technical support service.
4. Set up the MX-entries for each domain from which you will send emails. Do this even if you are not planning to receive any emails for this domain. You should at least be able to properly receive and process messages about failures when sending your emails and postmaster@ address. If in the sender’s address or in the SMTP-envelope you use domains without MX-entries, this will undermine email delivery. Some recipients validate sender addresses from SMTP-envelopes, which is why the server should have no failures when sending emails to this address.
5. Set up the SPF-record. SPF allows you to specify which servers can send email on behalf of the domain. SPF is configured for the address used in envelope-from (SMTP-envelope).
6. Set up DKIM:
6.1 Generate a DKIM key pair and publish a publicly available key;
6.2 Don’t forget to configure a DKIM signature for all outgoing emails (more on this later).
DKIM makes it possible to sign the emails sent on behalf of a certain domain. DKIM is currently a must-have technology, as it is the basis for others — FBL, DMARC, and such services as Postmaster Mail.ru.
7. Publish the DMARC policy. DMARC specifies how SPF and DKIM should be used and helps prevent phishing on behalf of the domain through a restriction policy. It also makes it possible to receive all information about attempts to violate your policy via structured reports. DMARC is not yet accepted as an official standard, but it is already supported by the major players in the email market: Mail.Ru, Google, Yahoo and Microsoft. It is an easily implemented and highly effective solution.
So now we can try to set up the email server.
8. In the Mail Transfer Agent settings (email server) configure the server’s hostname, which is passed via HELO command. This name should match the canonical server name from the PTR-record (server.example.com). It often happens that names such as localhost.localdomain are used in HELO by default, and many email servers do not accept emails with this HELO. Configure the SPF-record for the name from HELO. This is an RFC 7208 requirement enabling delivery of service emails with an empty address of the SMTP-envelope.
9. Enable DKIM support in your email server. Some servers have built-in support, and in others support can be implemented using free software (for example, OpenDKIM), for Microsoft Exchange / IIS SMTP there are inexpensive and free plug-ins. During configuration do not confuse DKIM (RFC 4871 / RFC 6376) and DomainKey (RFC 4870). DomainKey is a legacy standard which did not gain acceptance. It is not necessary to support it. DKIM can be found from the DKIM-Signature in email headers. For the email header and body signature it is best to use relaxed mode.
10. Set up the postmaster box.
11. Avoid configurations which cause unauthorized email relaying, otherwise all your work would be in vain, and the server’s reputation will be tarnished.
12. To test server configuration send emails to the major email services using a standard mail client, for example, Thunderbird. Check the headers of the emails received. Verify:
12.1 correctness of HELO;
12.2 presence and origin of SPF-, DKIM- and DMARC-verifications on the servers supporting DMARC;
12.3 that unauthorized relaying through your server is not possible;
12.4 that postmaster@ and addresses for DMARC and FBL reports receive emails
12.5 that server correctly generates reports about failure of email delivery.
13. Sign on to FBL (FeedBack Loop). FBL is provided by many large email services. The subscription will allow you to obtain information about all spam reports for emails from your server.
Now We Can Set Up Mailing Lists / Write Scripts To Send Letters. Let’s Start.
14. Process messages about failure to deliver emails. Remember that there are two ways to receive an error message: in an SMTP-session (in this case a message about delivery failure may be generated by your server) or via an email with a report about delivery failure (NDR) from the recipient server. You should process both cases. You must also respond and exclude users from mailing lists in the event of recurring or permanent delivery errors to their addresses. A large number of invalid recipients can lead to a lower reputation and/or to graylisting.
15. Install SMTP MAIL FROM address: (envelope sender) — sender’s address used on the SMTP level. It need not match the sender’s address specified in the From: email header. DMARC will work correctly if the From: address and the envelope sender address are from the same domain. And naturally the email should have a valid DKIM-entry for this domain and pass SPF verifications. An incorrect sender address in the envelope is the most common and serious error in mailing lists from various web-sites.
16. In From: and MAIL FROM addresses you should not use public mail addresses, because such messages will not pass SPF/DKIM authentication within DMARC. Use addresses from your own domain.
17. Specify correct encoding in Content-Type headers for all text parts of the email. Make sure that the real text encoding matches that specified in the header. We recommend using the same encoding in all headers and parts of the email. Currently UTF-8 is the recommended and the most common text encoding. If the encoding is not specified (or is specified incorrectly), the text may be displayed differently by different services and mail software.
18. Create the headers From:, Message-ID: and Date: directly in the script for email sending. Make sure that these headers, especially Date:, are generated in the correct format. According to the standards these headers are generated together with the email text. If they are absent or formatted incorrectly, headers may be added by one of the transit servers, which can ruin the integrity of the DKIM-signature.
19. Make sure that in all service or other headers, including Subject, names of files attached (Content-Type and Content-Disposition), characters different from ASCII, particularly Cyrillic, are encoded in quoted-printable or base64. According to the standards, email headers cannot contain non-encoded 8-bit characters; some servers do not accept such emails. If you’re oriented towards foreign subscribers, the email should not contain non-encoded 8-bit characters.
20. Limit the string length and use correct string terminators.
20.1 The email string should not contain more than 998 8-bit characters. If you are using UTF-8, one displayed character can be represented by several octet bytes, so try to make the string text shorter or encode the text in base64.
20.2 The correct string terminator in email is CRLF (characters with codes 13 and 10). The standard string terminator for Unix is LF. If the script or MTA does not replace string terminators, the email may be displayed incorrectly, or part of the email may be cut. Errors may also be caused by duplicate terminator replacement. Specify in the documentation or perform a test to determine whether the string endings should be re-encoded in the configuration that you use.
20.3 String terminators should not separate UTF-8 characters; this will help avoid a situation in which the terminator is added to a long string between two octets of a Cyrillic character. For this reason, the text should be laid out before creating an email.
Encoding text parts in base64 makes an email larger by a third, but it helps to avoid problems related to string terminators in text parts.
21. Try to make the email structure relatively simple, and avoid deep nesting of components (i.e. including one component in another). If an email contains more than one multipart of each type (one multipart/mixed, one multipart/alternative and one multipart/related), the message is most likely is formatted incorrectly.
22. Fill in Content-Disposition of every part as an attachment (shown separately from the message) or inline (an item shown as part of the message, e.g. a picture within the text). It often happens that an attached document is marked as inline, though it should be downloaded by a user, or a logo in an email text is marked as an attachment, though it should not be displayed in the attached files list.
23. Create a text (plaintext) part of the email as an alternative to the HTML-part. Remember that the user may read the message using a mobile phone, for example. It is not always possible to transform HTML-content into accurate and beautiful text. If you add a text part to a letter, you can be confident that the text will be displayed as you want it to.
24. Test how your script operates using test mailboxes from different email services. Remember to download an original version of the incoming letter and check the following:
24.1 correctness of envelope sender;
24.2 validation through SPF-, DKIM- and DMARC-verifications;
24.3 presence and correctness of text encoding in Content-Type headers for all text parts (including HTML);
24.4 in HTML-parts — correctness of encoding specified in metatags (if any);
24.5 display of different characters, including Cyrillic, in the text HTML-part and file names (if you plan to send the files);
24.6 presence of correct Content-Disposition and Content-Transfer-Encoding for every email part (except multipart);
24.7 display of pictures and attachments;
24.8 correctness of string terminators;
24.9 correct separation into strings and display of long texts in HTML and plain text parts;
24.10 Make sure that From:, Date: and Message-ID headers are created by your server, signed with a DKIM-signature and are valid.
Now We Can Turn To Email Layout.
Recommendations For The Layout Designer
25. Keep it simple. Remember that email web-services have to filter out some tags and attributes in order to ensure security, and all of them do it differently. Avoid using classes and minimize the use of styles, especially for positioning. Good old-fashioned layout in tables is the best option.
26. Do not try to create a layout based on the web-interface of a certain email server. Users often set up forwarding and mail fetchers or read emails in special software.
27. Think about user. Users may open your email on a large retina-display or on an old mobile phone on top of Mt. Everest, trying to find a GPRS-connection. So, emails should load quickly, but look great even if images are disabled.
28. No strings written with a small font. These annoy people and arouse suspicion. Don’t be afraid to show users that you know and respect their rights and your responsibilities.
29. Choose the most suitable way to include pictures. The following options are available, and each of them has its own pros and cons.
29.1 External images: this is the best option in terms of sending speed. External images are lightweight and do not complicate the email structure. But many applications and email services require a user’s permission in order to display such images. The server hosting them must also be reliable enough so that the message will not freeze if the image is unavailable. Use external images when images are optional.
29.2 Inline-images. These images are contained in email. They complicate the email structure and make it larger, but can be fairly large and usually are displayed by default.
29.3 Data URI. The image content is inserted directly into a tag. These do not complicate the email structure, almost always are displayed in email web-services and sometimes even when images are disabled. They are generally limited in size and can be used only for small icons.
29.4 Font images. Unicode character images from standard fonts or custom fonts. The unique characteristic is that they can scale up with the text. They weigh almost nothing. They may be displayed incorrectly, depending on the version of the operating system, updates installed, devices, the email software or service.
30. Make sure you specify the protocol (http://, https:// or mailto:) in all URIs; using a URI without a protocol, for example “//example.com/” is not allowed and can cause an email to freeze in some email applications because of attempts to connect with the network; using other protocols is not recommended because they may be filtered out.
31. Test display of every new layout on different devices in different email web-services prior to sending it to real users.
What should you do if, despite following all these recommendations, users still do not receive emails? Contact the technical support service of the email recipients — usually such problems can be resolved quickly. Regarding problems with delivery to addresses serviced by Mail.Ru server, you can write to abuse@corp.mail.ru. Describe the problem in detail — it is very helpful to include error messages, messages about delivery failure, server logs and other technical information.
Unfortunately, the article format doesn’t allow to include more details on each recommendation or specific configuration instructions, but if you have any technical questions, I would be happy to answer them in comments.
Originally published at team.mail.ru on December 12, 2016.
