This is a quick recap of why I'm sad about SMTP encryption. It explains how TLS certificate verification in SMTP is useless even if you force it.
SMTP
SMTP is the protocol that mail servers talk between them to deliver mail. Standardized in 1982 it used to be, unsurprisingly, 100% plaintext.
It works like this: a mail server mail.company.com has an email for alice@example.com. It does a DNS lookup for a MX record:
;; ANSWER SECTION:
example.com. 300 IN MX 10 smtp.example.com.
;; ADDITIONAL SECTION:
smtp.example.com. 300 IN A 1.2.3.4
Then it opens a connection to 1.2.3.4, exchange greetings, and deliver the email.
S: 220 smtp.example.com ESMTP Postfix
C: HELO mail.company.com
S: 250 Hello mail.company.com, I am glad to meet you
C: MAIL FROM:<bob@company.com>
S: 250 Ok
C: RCPT TO:<alice@example.com>
S: 250 Ok
...
SMTP is also used by local email clients to submit email, but today we're not talking about that.
STARTTLS
Then around 2000 people realized that plaintext was a bad idea and started to retrofit encryption on top of protocols. Enter STARTTLS [RFC3207].
The idea is simple: upon connection the server announces its support for STARTTLS and if the client supports it too the connection is "upgraded". That is, a TLS handshake is performed and the connection resumes over the encrypted channel.
It looks like this on the wire — keep in mind that here "client" and "server" are both mail servers:
S: 220 smtp.example.com ESMTP Postfix
C: HELO mail.company.com
S: 250 Hello mail.company.com, I am glad to meet you
S: 250 STARTTLS
C: STARTTLS
S: 220 Go ahead
C & S: <start a TLS session>
C [TLS]: EHLO mail.company.com
S: 250 Hello mail.company.com, I am glad to meet you
C: MAIL FROM:<bob@company.com>
S: 250 Ok
C: RCPT TO:<alice@example.com>
S: 250 Ok
...
The obvious problem
STARTTLS has a glaring problem: it's negotiated over the plaintext channel.
The S: 250 STARTTLS line is sent on the clear so an attacker performing a MitM attack can just block it, prevent it from ever reaching the client. At that point the client will simply go ahead with unencrypted SMTP, unaware that the server supports TLS, and the server will think it's the client that came short in supporting it.
In browser world, it's as if you always connected over HTTP and relied on the 301 redirect to switch to HTTPS. An attacker can do a SSL stripping attack where they just answer to your HTTP query. It's also what HSTS is designed to prevent.
So in its normal configuration STARTTLS is not effective against an active attacker, whom can just downgrade the connection at will.
The insidious problem
At this point you might think you can make a choice: "I'll require encryption on all my connections and accept the tradeoff of only receiving from and delivering to servers that support STARTTLS". Or, "I'll make a whitelist of known domains for which I require STARTTLS because I know they support it".
You actually can't.
The problem is in certificate verification: what hostname do you verify a certificate against? That is, what name do you require on the certificate to make sure you are not talking with an attacker?
The answer is "whatever the target of the MX record is". And this makes certificate validation—and signed certificates—completely useless. (Indeed, all mail servers will accept expired, self–signed certificates.)
First, let's see why we can't verify against the domain portion of the email address. Take my email: it's @filippo.io but it's handled by FastMail. FastMail doesn't have a certificate for filippo.io. How it works instead is that a client will perform a MX lookup
;; ANSWER SECTION:
filippo.io. 300 IN MX 20 in2-smtp.messagingengine.com.
filippo.io. 300 IN MX 10 in1-smtp.messagingengine.com.
and then verify the certificate against inX-smtp.messagingengine.com (FastMail).
If you ask me it's silly, since whoever is able to receive mail for your domain would be able to get a DV certificate issued for it by any CA. But it doesn't happen and this is the Internet we are stuck with.
So, verification against the MX target. What does this mean for security?
It means that verifying certificates is pointless. The MX record lookup is unencrypted. A MitM'ing attacker can simply falsify the MX like this
example.com. 300 IN MX 10 smtp.attacker.com.
and then present a valid shiny certificate for smtp.attacker.com. Over.
Opportunistic encryption
STARTTLS can't be secured in any way against an active attacker. So, what is it good for? Well, it's opportunistic encryption and it's not worthless.
Opportunistic encryption changes the attack requirement from simple passive observation to risky active interception. It stops dragnet collection that we know too well some agencies perform.
So, bring it on with STARTTLS, just don't bother with getting a CA issued certificate and know that a targeted MitM–capable attacker can totally intercept your mail on the wire.
By the way, keep in mind that SMTP encryption is just protection on transit: it doesn't hide the email content from any mail server in the path and it does not do anything to authenticate the sender.
DNSSEC
There is actually a solution and it's DNSSEC. SMTP is the use case where it really shines!
Mail servers can perform DNSSEC verification to make sure the MX is not tampered with, and then use DANE to retrieve the intended certificate fingerprint. This would give SMTP a fighting chance against an active attacker.
It also degrades gracefully, because it's possible to securely determine if a domain is not signed with DNSSEC and fallback to the old no-verification scheme without risk of downgrade.
EDIT: @hanno brought this EFF initiative to my attention: EFForg/starttls-everywhere. It attempts to solve the same problem with a more powerful whitelist approach, listing allowed MX records and TLS policies for each domain. It does not really scale, but sounds like a good stopgap.
Conclusion
So this is it. The Internet is terrible and until DNSSEC sees wide deployment SMTP will have at best opportunistic encryption.
A practical note on active attacks: MitM on a server-to-server connection is not a common capability but it's very possible for a determined attacker. For example BGP hijacks happen frighteningly often and there are agencies with taps on the Internet backbones.
Still, STARTTLS is better than allowing dragnet surveillance, so please, please support it. You don't even have to get a signed certificate! There are no excuses.
For more sadness, follow me on Twitter.