Playing with kernel TLS in Linux 4.13 and Go

Linux 4.13 introduces support for nothing less than... TLS!

The 1600 LoC patch allows userspace to pass the kernel the encryption keys for an established connection, making encryption happen transparently inside the kernel.

The only ciphersuite supported is AES-128-GCM as per RFC 5288, meaning it only supports TLS version 1.2. Most modern TLS connections on the Internet use that.

The kernel only handles the record layer, that is, it only takes care of encrypting packets. Handshake, key exchange, certificate handling, alerts and renegotiation are left out of kernelspace. The userspace application, like OpenSSL, will do all that and then delegate to the kernel once the keys are established.

Moreover, only encryption is supported, not decryption. This wasn't clear to me until I failed to find the TLS_RX constant.

These limitations are very good to contain complexity and attack surface, but they mean that kTLS won't replace any userspace complexity as you still need a TLS library to do the handshake, for all other cipher suites, and for the receiving side of the connection. That makes kTLS purely a performance feature.

Keys are passed to the kernel with a setsockopt(2) call on the TCP socket. Once that call is made everything written to that socket is transparently encrypted. A TLS record is made for each send(2) call unless a flag is used to request buffering. Alerts and any other messages which are not application data are sent via CMSG.

The main motivation seems to be to allow use of sendfile(2) on TLS connections. sendfile(2) allows data to be transferred from a file descriptor (like a file) to another (like a TCP connection) without paying the price of a copy round-trip through user space. If the kernel is handling TLS, sendfile(2) can be used also for encrypted connections. The original paper by Facebook claims a significant improvement in 99th percentile performance.

I also saw a mention of using BPF filtering rules on plaintext data, which is clever.

It seems all very sensibly executed, but I can't help being terrified nonetheless. First because even just the record layer of TLS 1.2 with AEAD has significant legacy baggage and attack surface, and secondly because any compatibility issue introduced by this code will add a dimension to the compatibility matrix.

Moreover, the end goal seems to be to do TLS offloading on dedicated hardware managed by kernel drivers, and poorly-implemented hard-to-update hardware is exactly why TLS 1.3 hasn't been deployed yet.

So yeah, not a huge fan. But of course, it wouldn't be me if I didn't fork the Go crypto/tls package to work with it.

To test kTLS I'll need a Linux VM running Linux 4.13. My favorite Linux distribution (Alpine) is not that cutting-edge with kernel versions, but one can of course trust Arch to have an up-to-date linux-mainline package. 3 painful hours of learning Arch and compiling Linux from AUR follow...

To enable TLS support make sure to compile with CONFIG_TLS.

[root@localhost ~]# uname -a
Linux localhost 4.13.0-mainline #1 SMP PREEMPT Wed Sep 6 01:04:02 BST 2017 x86_64 GNU/Linux

I tried toying with mkerrors.sh to get the TLS constants into the syscall package, but eventually gave up and redefined them in crypto/tls. There is a CL for golang.org/x/sys/unix to update to Linux 4.13, but it's compiled without kTLS.

The easiest place to hook kTLS into crypto/tls seemed to be (*halfConn).changeCipherSpec(). That's where encryption is enabled for that half (receiving or sending) of the connection. However that happened before sending the Finished handshake message, which we would have to send with a CMSG.

Instead, I added a hook at the end of both client and server handshake. The new function (*Conn).enableApplicationDataEncryption() checks that the cipher is a *fixedNonceAEAD (i.e. AES-GCM) with the right key length, and that the connection is a *net.TCPConn, and then invokes kTLSEnable() with all the key material.

kTLSEnable constructs the tls_crypto_info structure as per the docs, hoping Go won't add padding for alignment purposes. It then uses syscall.RawConn.Control and syscall.SetsockoptString to make the setsockopt syscalls to pass it to the kernel. There's a lot of unsafe involved, but no need for cgo (I just redefine the types manually).

One of the quirks of AES-GCM in TLS 1.2 is that it has 4 bytes of implicit IV and 8 bytes of explicit IV (counter). Linux calls the explicit part iv and the implicit part... salt. Ok. I guess.

For a moment I thought it would be enough to then switch the cipher internally to unencrypted, but while it's not documented, for sendfile(2) to work the data must be sent with no framing at all. Even unencrypted TLS packets are framed. So as a last hack I added a dummy kTLSCipher that strips the record header instead of encrypting the record before sending it on the wire.

I ran a simple HTTPS web server with net/http, loaded a page on Chrome, and instead of causing a kernel panic...

[root@localhost ~]# go run https_server.go
2017/09/06 16:21:39 kTLS: enabled
2017/09/06 16:21:39 kTLS: sent 33 bytes of plaintext to the kernel
2017/09/06 16:21:39 kTLS: enabled
2017/09/06 16:21:39 kTLS: dropped alert on the floor
2017/09/06 16:21:39 kTLS: sent 0 bytes of plaintext to the kernel
2017/09/06 16:21:39 kTLS: enabled
2017/09/06 16:21:39 kTLS: sent 33 bytes of plaintext to the kernel
2017/09/06 16:21:39 kTLS: sent 107 bytes of plaintext to the kernel
2017/09/06 16:21:39 kTLS: sent 37 bytes of plaintext to the kernel

💥

Future work includes getting the io.WriterTo-based sendfile(2) interface upgrade to work across the *tls.Conn, and running the various TLS test suites and fuzzers against kTLS.

You can see the code here. Please don't use it.

But maybe follow me on Twitter!