OpenBSD manual page server

Manual Page Search Parameters

STARTTLS(8) System Manager's Manual STARTTLS(8)

starttlsESMTP over TLS/SSL

STARTTLS is an ESMTP option, defined in RFC 3207, which is used to conduct ESMTP transactions over TLS circuits. This is used to increase the security of mail server transactions. As of version 8.11, sendmail(8) has supported the use of TLS to protect ESMTP communications.

STARTTLS allows for the combination of several security solutions for MTA (mail transport agent) level services through the TLS suite. These security features include:

Encryption is used to protect data from passive monitoring. An attacker would have to recover the encryption key used to decode the transmitted data.
Hash algorithms are used to ensure the integrity of the transmitted data, and alternatively the timestamp, protecting against a replay attack. This protects data from modification in transit.
The use of public key encryption allows for the strong authentication of either, or both, communicating parties. This can be used to allow for select features, such as relaying, to be controlled more securely.

A new ESMTP option, STARTTLS, has been added. This is presented by the server when an ESMTP session is initiated. The client then begins the TLS portion of the ESMTP session by issuing the command “STARTTLS”. The remaining portion of the ESMTP session occurs over a TLS channel.

This example assumes you are creating your own self-signed certificates for use with sendmail and STARTTLS. If you have an existing private key and you simply wish to generate a new certificate (for example, if your old certificate has expired), see the section entitled Creating a certificate with an existing private key.

For the purposes of this example the certificates will be stored in /etc/mail/certs, though it is possible to use a different directory if needed. If this directory does not already exist, you must create it:

# mkdir /etc/mail/certs

Next, you must generate a DSA parameter set with a command like the following:

# openssl dsaparam 1024 -out dsa1024.pem

This would generate DSA parameters for 1024-bit DSA keys, and save them to the file dsa1024.pem.

Once you have the DSA parameters generated, you can generate a certificate and unencrypted private key using the command:

# openssl req -x509 -nodes -days 365 -newkey dsa:dsa1024.pem \
  -out /etc/mail/certs/mycert.pem -keyout /etc/mail/certs/mykey.pem

You may adjust the lifetime of the certificate via the -days parameter (one year in this example).

Make sure to leave the private key files unencrypted, using the -nodes option. Otherwise, sendmail(8) will be unable to initiate TLS server functions.

You can verify that the newly-generated certificate has correct information with the following command:

# openssl x509 -in /etc/mail/certs/mycert.pem -text

If you don't intend to use TLS for authentication (and if you are using self-signed certificates you probably don't) you can simply link your new certificate to CAcert.pem.

# ln -s /etc/mail/certs/mycert.pem /etc/mail/certs/CAcert.pem

If, on the other hand, you intend to use TLS for authentication you should install your certificate authority bundle as /etc/mail/certs/CAcert.pem.

At this point, you no longer need the dsa1024.pem file and it can be removed.

# rm dsa1024.pem

Because the private key files are unencrypted, sendmail(8) is picky about using tight permissions on those files. The certificate directory and the files therein should be readable and writable only by the owner (root). A simple way to ensure this is to run the following:

# chmod -R go-rwx /etc/mail/certs

This example assumes you already have an existing private key, /etc/mail/certs/mykey.pem. You can generate a new certificate based on this key using the command:

# openssl req -x509 -new -days 365 -key /etc/mail/certs/mykey.pem \
  -out /etc/mail/certs/mycert.pem
# chmod 600 /etc/mail/certs/mycert.pem

You may adjust the lifetime of the certificate via the -days parameter (one year in this example).

The global sendmail configuration files, /etc/mail/ and /etc/mail/ must be modified to support TLS functionality. An example .mc file which has entries for TLS options, such as certificates, is available as /usr/share/sendmail/cf/

The pertinent options are:

  • confCACERT
  • confSERVER_KEY
  • confCLIENT_KEY

By default, the directory /etc/mail/certs, defined by CERT_DIR, is used to store certificates, and the server will use the same certificates both as a client (outgoing mail) and as a server (for incoming mail). This can be changed by having different entries for the respective roles.

The next step is to edit the .mc files your and files are generated from. First, change to the directory where your .mc files are stored. You will need to make TLS-enabled versions of the following files: and

# cd /usr/share/sendmail/cf

# cp
# cp

You must then edit, and to add the following lines after the “VERSIONID” definition (the actual placement within the file is not critical as long as it is after the “divert(0)dnl” line).

define(`CERT_DIR',        `MAIL_SETTINGS_DIR`'certs')
define(`confCACERT_PATH', `CERT_DIR')
define(`confCACERT',      `CERT_DIR/CAcert.pem')
define(`confSERVER_CERT', `CERT_DIR/mycert.pem')
define(`confSERVER_KEY',  `CERT_DIR/mykey.pem')
define(`confCLIENT_CERT', `CERT_DIR/mycert.pem')
define(`confCLIENT_KEY',  `CERT_DIR/mykey.pem')

Now that you have the TLS-enabled versions of the .mc files you must generate .cf files from them and install the .cf files in /etc/mail.

# make

# cp /etc/mail/
# cp /etc/mail/

Finally, restart sendmail with the new configuration by sending it a SIGHUP.

# kill -HUP `head -1 /var/run/`

Note that those are backticks and not single quotes in the example above.

After having installed the certificates and configuration files and having restarted the sendmail daemon, a new option should be presented for ESMTP transactions, STARTTLS. You can test this by connecting to the local host and issuing the “EHLO” command.

# telnet localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 localhost ESMTP Sendmail 8.12.1/8.12.1 ready
EHLO localhost

After typing you should receive something like the following back.

250-localhost Hello localhost [IPv6:::1], pleased to meet you
250 HELP

You should see “STARTTLS” listed along with the other options. If so, congratulations, sendmail will now use TLS to encrypt your mail traffic when the remote server supports it. If not, check /var/log/maillog to see whether sendmail has reported any security problems or other errors.

The most obvious use of a cryptographically enabled sendmail installation is for confidentiality of the electronic mail transaction and the integrity checking provided by the cipher suite. All traffic between the two mail servers is encrypted, including the sender and recipient addresses. TLS also allows for authentication of either or both systems in the transaction.

One use of public key cryptography is for strong authentication. We can use this authentication to selectively relay clients, including other mail servers and mobile clients like laptops. However, there have been some problems getting some mail clients to work using certificate-based authentication. Note that your clients will have to generate certificates and have them signed (for trust validation) by a CA (certificate authority) you also trust, if you configure your server to do client certificate checking.

The use of the access map (usually /etc/mail/access), which is normally used to determine connections and relaying, can also be extended to give server level control for the use of TLS. Two new entries are available for TLS options:

contains the status of the level of verification (held in the macro {verify})
the strength of the encryption (in the macro {cipher_bits})

VERIFY can also accept the argument for {cipher_bits}. Here are a few example entries that illustrate these features, and the role based granularity as well:

Require strong (256-bit) encryption for communication with this server: ENCR:256

For a TLS client, require verification and a minimum of 128-bit encryption: VERIFY:128

Much more complicated access maps are possible, and error conditions (such as permanent or temporary, PERM+ or TEMP+) can be set on the basis of various criteria. This allows you fine-grained control over the types of connections you can allow.

Note that it is unwise to force all SMTP clients to use TLS, as it is not yet widespread. The RFC document notes that publicly referenced SMTP servers, such as the MX servers for a domain, must not refuse non-TLS connections. However, restricted access SMTP servers, such as those for a corporate intranet, can use TLS as an access control mechanism.

One often forgotten limitation of using TLS on a mail server is the payload of the mail message and the resulting security there. Many virus and worm files are now distributed via electronic mail. While the mail may be encrypted and the servers authenticated, the payload can still be malicious. The use of a good content protection program on the desktop is therefore still of value even with TLS at the MTA level.

Because sendmail with TLS can only authenticate at the server level, true end-to-end authentication of the mail message cannot be performed with only the use of STARTLS on the server. The use of S/MIME or PGP e-mail and trustworthy key hierarchies can guarantee full confidentiality and integrity of the entire message path.

Furthermore, if a mail message traverses more than just the starting and ending servers, there is no way to control interactions between the intervening mail servers, which may use non-secure connections. This introduces a point of vulnerability in the chain.

Additionally, SMTP over TLS is not yet widely implemented. The standard, in fact, doesn't require it, leaving it only as an option, though specific sites can configure their servers to force it for specific clients. As such, it is difficult to foresee the widespread use of SMTP using TLS, despite the fact that the standard is, at the date of this writing, over two years old.

Lastly, interoperability problems can appear between different implementations.

mail(1), openssl(1), afterboot(8), sendmail(8), ssl(8)

P. Hoffman, SMTP Service Extension for Secure SMTP over Transport Layer Security, RFC 3207, February 2002.

TLS features first appeared in sendmail 8.11.

October 16, 2012 OpenBSD-5.6