Securing SMTP sessions using the STARTTLS extension

SMTP sessions conducted over a standard TCP/IP channel are vulnerable to eavesdropping because the unencoded transmission can be easily intercepted. To protect SMTP communications, servers can use transport-layer security (TLS), more commonly known as SSL encryption, to provide privacy and authentication.

Some servers support SSL for SMTP communications by sending and receiving SMTP traffic through the SSL port (port 465 by default) only. However, because this requires that both the sending and receiving servers support SMTP over SSL, this solution isn't always practical.

To provide SSL security for SMTP transfers over TCP/IP, Domino® supports the use of negotiated SSL. In a negotiated SSL scheme, the sending and receiving hosts each use the SMTP STARTTLS extension, defined in RFC 2487, to signal their readiness to negotiate an SSL connection. The receiving server displays the STARTTLS keyword in response to the sending server's EHLO command. The sending server issues the STARTTLS command to request the creation of a secure connection. After the initial TLS handshake completes successfully, the two parties proceed to set up an SSL channel between them. Both the sending and receiving server must possess SSL certificates.

Supporting STARTTLS for outbound SMTP sessions

A Domino® server configured to use negotiated SSL for outbound mail connects to the receiving server's SMTP TCP/IP port (port 25 by default). If the initial SMTP response from the receiving server indicates that it supports the STARTTLS extension, Domino® issues the STARTTLS command to request the use of SSL to encrypt the rest of the session.

If the receiving server did not advertise support for STARTTLS in response to the Domino® server's EHLO command, the sending Domino® server continues with an unencrypted SMTP TCP/IP session.

To enable outbound STARTTLS support, set the SMTP outbound TCP/IP port status to: Negotiated SSL.

Supporting STARTTLS for inbound SMTP sessions

You can configure Domino® to support the STARTTLS command for inbound SMTP transactions. When a Domino® SMTP server is set to use negotiated SSL for inbound sessions, the server advertises support for STARTTLS in response to EHLO commands the TCP/IP port receives from connecting hosts. A connecting host can then issue the STARTTLS command to request an encrypted session.

If Domino® is configured to require STARTTLS for SMTP sessions over TCP/IP and a connecting host cannot meet this demand, no mail is sent over the connection.

To enable inbound STARTTLS support:

  • Enable the SMTP listener task.
  • Enable the SMTP inbound TCP/IP port.
  • Enable the STARTTLS ESMTP extension. This causes Domino® to advertise STARTTLS as one of its supported extensions in the ESMTP EHLO greeting response.
  • (Optional) Enable name-and-password authentication for the SSL port. Although SMTP sessions that use negotiated SSL are conducted over the Domino® TCP/IP port, Domino® uses the authentication options you set for the server's SSL port to determine how to handle name-and-password arguments.

Requiring name and password authentication for SMTP STARTTLS sessions

Enabling ESMTP support for negotiated SSL allows a server to accept requests to use SSL over TCP/IP from remote servers that connect anonymously. However, not all inbound connections are anonymous. A connecting SMTP server may be configured to send Domino® a name and password by means of the ESMTP AUTH command.

To support connections from SMTP clients that send a name and password during a negotiated SSL session, set the value of the Name & password field for the SMTP inbound SSL port to Yes. You do not have to enable the SSL port. If the SSL port does not support name-and-password authentication, the Domino® SMTP server rejects the AUTH command from the remote server and returns an error indicating that the command is not implemented.

Even though Domino® receives the AUTH command over the TCP/IP port, Domino® uses the SSL name-and-password authentication settings to determine whether to accept the AUTH request because it receives the command in the context of an SSL session. The Name & password authentication setting for the TCP/IP port is ignored.