What is StartTLS? How it works, uses, and security risks

The StartTLS command (also known as “Op­por­tunis­tic TLS”) extends the Transport Layer Security (TLS) protocol to encrypt the com­mu­ni­ca­tion of in­for­ma­tion using the TLS protocol.

How does StartTLS work?

As a protocol extension for TLS, StartTLS offers the major advantage that en­cryp­tion does not impede com­mu­ni­ca­tion with clients that do not support the method. Thus, there are far fewer com­pat­i­bil­i­ty issues with StartTLS. However, if a server refuses TLS, a mail program (client) must know how to proceed with the data. Another advantage is the mutual ne­go­ti­a­tion of en­cryp­tion, which makes automated processes un­nec­es­sary without the admin’s or user’s in­ter­ven­tion in case of a com­mu­ni­ca­tion failure.

In the StartTLS process, a con­nec­tion always starts un­en­crypt­ed, specif­i­cal­ly on the port des­ig­nat­ed for plaintext. Only after executing the StartTLS command does the protocol negotiate en­cryp­tion with the client, without es­tab­lish­ing a new con­nec­tion. Thanks to StartTLS, if there’s a com­mu­ni­ca­tion error, no port change is required, and the client can simply rely on the StartTLS procedure provided by the email server.

Close

Example of en­crypt­ing an email

An email is to be encrypted using the TLS method. The StartTLS protocol is selected to seam­less­ly integrate the en­cryp­tion method into normal operation. You usually find the cor­re­spond­ing option in the account con­fig­u­ra­tion or account settings of your email client. After sending the email, com­mu­ni­ca­tion begins:

1. The client sends an EHLO request to inquire about the en­cryp­tion methods.
2. The mail server responds with “250-STARTTLS,” in­di­cat­ing that it accepts StartTLS.
3. The client sends “STARTTLS” to initiate en­cryp­tion.
4. The server responds with the status code “220 OK”.

Areas of ap­pli­ca­tion for StartTLS

StartTLS is primarily used as a protocol extension in com­mu­ni­ca­tion via email for the protocols SMTP, IMAP, and POP. HTTP has its own procedure (specified in RFC 2817) that is very similar to StartTLS. Today, however, the use of HTTPS (RFC 2818) is more common. Besides the mentioned email protocols, StartTLS can at least initiate the en­cryp­tion process in the following protocols:

• LDAP (RFC 4511)
• FTP (RFC 4217)
• XMPP (RFC 6120)
• NNTP (RFC 4642)

With internet providers, StartTLS is by far the most popular email en­cryp­tion method because it allows many different domains and cer­tifi­cates on one server. However, the method is con­tro­ver­sial because some private details like the IP address are initially un­en­crypt­ed and Man-in-the-Middle attacks are possible, as StartTLS can be over­writ­ten unnoticed by the provider.

Ad­van­tages and dis­ad­van­tages of StartTLS

Dis­ad­van­tages become apparent in the StartTLS procedure in terms of security software. Firewalls must analyze the procedure at the ap­pli­ca­tion layer to dis­tin­guish encrypted from un­en­crypt­ed data. The same applies to proxies, which prefer to work with port dif­fer­en­ti­a­tion – however, with StartTLS, the ports are not switched. This makes the cor­re­spond­ing caching difficult or not executed at all.

When it comes to data pro­tec­tion, StartTLS is not without concerns. Most email programs use the option “TLS if available,” which means users are unaware if the con­nec­tion to the mail server is encrypted or not. Ad­di­tion­al­ly, the risk of a man-in-the-middle attack increases as the network operator can simply filter out the StartTLS extension, getting the chance to log the data exchange. If the StartTLS command is not executed, the data is com­mu­ni­cat­ed un­en­crypt­ed – and users generally do not notice this.

Therefore, it’s rec­om­mend­ed to first conduct a careful test to ensure that the server is indeed capable of StartTLS. Only then should the procedure be used by default. If encrypted com­mu­ni­ca­tion fails, an external problem must exist.

How can I test StartTLS?

TLS protocols use different ports depending on the provider. However, because StartTLS can be easily embedded in the con­nec­tion, making port switching un­nec­es­sary, the en­cryp­tion method can be tested with a suitable console. This shows which port the re­spec­tive provider uses for StartTLS. The following command can be used (e.g., in Netcat) to check if a mail server accepts StartTLS as an en­cryp­tion method during normal operation:

$ nc smtp.ionos.com 587
220 mail.example.com (mreue011) Nemesis ESMTP Service ready
ehlo test.local
250-mail.example.com Hello test.local [80.187.86.70]
250-8BITMIME
250-SIZE 141557760
250 STARTTLS
STARTTLS
220 OK

The command “STARTTLS” should appear here to enable en­cryp­tion. You can also see that private data, such as the IP address, is trans­mit­ted un­en­crypt­ed.

For OpenSSL, the command looks as follows:

$ openssl s_client -starttls smtp -connect smtp.ionos.com:587
CONNECTED(00000005)
[…]
SSL handshake has read 4650 bytes and written 1659 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Server public key is 3072 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 STARTTLS

For SMTP, au­then­ti­ca­tion is not as straight­for­ward. The “AUTH PLAIN” method usually provides a solution, where cre­den­tials must be encoded in Base64. A code can be generated with the Perl command $ perl -MMIME::Base64 -e 'print encode_base64("\000user-ju\000secret"), which is then given to the SMTP server (AUTH PLAIN [Code]). If the server responds with Authentication succeeded (with code), the StartTLS test was suc­cess­ful. Sub­se­quent­ly, further mail commands can be executed.