TLS 1.3 Handshake, in depth: the message flow, what each message proves, the client state machine, a runnable toy handshake and the attacks it stops

Three problems in one round trip

The handshake has to solve three problems at once. First, key agreement: two parties who share no secret must derive one over a network that an attacker can read. TLS 1.3 uses ephemeral Diffie-Hellman for this, usually X25519 or P-256; the mathematics is covered in Diffie-Hellman, in depth. Because the key pairs are generated fresh for each connection and thrown away, stealing the server's long-term key later does not decrypt recorded traffic. That property is forward secrecy, and TLS 1.3 made it mandatory by removing static RSA key transport.

Second, authentication: Diffie-Hellman alone happily agrees a key with whoever is in the middle. The server therefore signs the conversation with the private key bound to its certificate, and the client checks the certificate chain against its trust store and the requested hostname. Third, integrity of the negotiation: an attacker who cannot read traffic might still edit the plaintext hello messages to push both sides to weaker options. Both sides therefore keep a running hash of every handshake message, the transcript hash, and the signature and the Finished messages are computed over it. Any edit to any message makes the two transcripts differ, and the checks fail.

The message flow

The client opens by guessing. Its ClientHello carries a random value, the cipher suites it supports, a supported_versions extension listing TLS 1.3, its signature algorithms, and, crucially, a key_share: a Diffie-Hellman public key for the group it expects the server to choose. Sending the key share in the first message, instead of waiting to negotiate a group, is what removed the extra round trip of TLS 1.2.

The server picks a cipher suite and group, generates its own key share, and replies with ServerHello. At that moment both sides can compute the shared secret, so every following server message is encrypted: EncryptedExtensions (things like the ALPN protocol), Certificate, CertificateVerify and Finished. The client verifies them, sends its own Finished, and can send application data immediately. If the client guessed the wrong group, the server answers with a HelloRetryRequest naming the group it wants, and the client sends a second ClientHello, costing one extra round trip.

What each message proves

It helps to read each message as a claim plus evidence.

The ranges in the last column are exact and easy to get wrong. CertificateVerify signs a fixed context string, 64 bytes of 0x20, the text TLS 1.3, server CertificateVerify, a zero byte, then the transcript hash up to and including Certificate. The context string stops a signature made for TLS from being replayed in another protocol that uses the same key. Note also that ServerHello's random field carries a downgrade sentinel: a TLS 1.3 server that is forced to negotiate TLS 1.2 writes a fixed marker into the last eight bytes, which a TLS 1.3 client detects and rejects.

The key schedule in one screen

Keys come from HKDF, a two-step function: Extract compresses input keying material into a pseudorandom key, and Expand stretches a key into as many labelled outputs as needed. TLS wraps Expand as HKDF-Expand-Label, which prefixes every label with tls13 and binds the output length and a context, usually a transcript hash, into the info field. The schedule is a chain of three extractions.

early     = HKDF-Extract(salt=0,                         ikm=PSK or 0)
handshake = HKDF-Extract(salt=Derive(early, "derived"),     ikm=ECDHE shared secret)
master    = HKDF-Extract(salt=Derive(handshake, "derived"), ikm=0)

c_hs = Derive(handshake, "c hs traffic", hash(CH..SH))
s_hs = Derive(handshake, "s hs traffic", hash(CH..SH))
c_ap = Derive(master,    "c ap traffic", hash(CH..server Finished))
s_ap = Derive(master,    "s ap traffic", hash(CH..server Finished))

finished_key = HKDF-Expand-Label(base_key, "finished", "", hash_len)
verify_data  = HMAC(finished_key, hash(transcript so far))

Two design points matter for understanding attacks. Each traffic secret is bound to the transcript at the point it is derived, so keys themselves change if any earlier message changed. And handshake keys and application keys are separate, so a flaw that leaks handshake traffic does not expose application data.

The handshake as a state machine

Most real-world handshake vulnerabilities were not broken cryptography but broken state machines: code that accepted a message in the wrong state or skipped a check when a message was missing. The 2015 SMACK research (state machine attacks) found TLS libraries that, for example, would accept a server that skipped the key-exchange messages entirely. The defence is to write the handshake as an explicit transition table and treat everything else as fatal.

TRANSITIONS = {
    ("WAIT_SH",       "server_hello"):         "WAIT_EE",
    ("WAIT_SH",       "hello_retry_request"):  "START",        # send a new ClientHello once
    ("WAIT_EE",       "encrypted_extensions"): "WAIT_CERT_CR", # or WAIT_FINISHED when resuming
    ("WAIT_CERT_CR",  "certificate_request"):  "WAIT_CERT",
    ("WAIT_CERT_CR",  "certificate"):          "WAIT_CV",
    ("WAIT_CERT",     "certificate"):          "WAIT_CV",
    ("WAIT_CV",       "certificate_verify"):   "WAIT_FINISHED",
    ("WAIT_FINISHED", "finished"):             "CONNECTED",
}

def step(state, msg_type, retried):
    nxt = TRANSITIONS.get((state, msg_type))
    if nxt is None:
        raise Alert("unexpected_message")
    if msg_type == "hello_retry_request" and retried:
        raise Alert("unexpected_message")      # at most one HRR per connection
    return nxt

A runnable toy handshake

The toy below performs the certificate-path handshake between an in-process client and server. It uses X25519 for the key share, Ed25519 in place of a certificate chain (the client pins the server's public key), and a correct HKDF-Expand-Label. Two assertions check the early secret and the first derived secret against the RFC 8448 test vectors. What it leaves out is deliberate: no record layer, no encryption of the server flight, no cipher suite negotiation and no real certificates, so never use it to protect anything. It requires the cryptography package.

import hashlib, hmac, struct
from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
from cryptography.hazmat.primitives import serialization

H, HLEN = hashlib.sha256, 32

def extract(salt, ikm):
    return hmac.new(salt, ikm, H).digest()

def expand_label(secret, label, context, length):
    full = b"tls13 " + label
    info = struct.pack(">H", length) + bytes([len(full)]) + full + bytes([len(context)]) + context
    out, block, i = b"", b"", 1
    while len(out) < length:
        block = hmac.new(secret, block + info + bytes([i]), H).digest()
        out, i = out + block, i + 1
    return out[:length]

def derive(secret, label, transcript):
    return expand_label(secret, label, H(transcript).digest(), HLEN)

EARLY = extract(b"\0" * HLEN, b"\0" * HLEN)
assert EARLY.hex().startswith("33ad0a1c607ec03b")                      # RFC 8448
assert derive(EARLY, b"derived", b"").hex().startswith("6f2615a108c702c5")

raw = lambda k: k.public_bytes(serialization.Encoding.Raw, serialization.PublicFormat.Raw)
msg = lambda kind, body: bytes([kind]) + len(body).to_bytes(3, "big") + body
CV_CTX = b"\x20" * 64 + b"TLS 1.3, server CertificateVerify\x00"

def finished(base, transcript):
    return hmac.new(expand_label(base, b"finished", b"", HLEN), H(transcript).digest(), H).digest()

def handshake(server_key, pinned, tamper=False):
    c_eph = X25519PrivateKey.generate()
    ch = msg(1, b"cr" + raw(c_eph.public_key()))
    # server side
    s_eph = X25519PrivateKey.generate()
    sh = msg(2, b"sr" + raw(s_eph.public_key()))
    hs = extract(derive(EARLY, b"derived", b""), s_eph.exchange(c_eph.public_key()))
    s_hs, c_hs_srv = derive(hs, b"s hs traffic", ch + sh), derive(hs, b"c hs traffic", ch + sh)
    ee, cert = msg(8, b"{}"), msg(11, raw(server_key.public_key()))
    cv = msg(15, server_key.sign(CV_CTX + H(ch + sh + ee + cert).digest()))
    sfin = msg(20, finished(s_hs, ch + sh + ee + cert + cv))
    if tamper:
        sh = msg(2, b"XX" + sh[6:])           # attacker edits ServerHello in flight
    # client side
    hs_c = extract(derive(EARLY, b"derived", b""), c_eph.exchange(s_eph.public_key()))
    c_hs = derive(hs_c, b"c hs traffic", ch + sh)
    if cert[4:] != pinned:
        raise ValueError("unknown server key")
    Ed25519PublicKey.from_public_bytes(cert[4:]).verify(cv[4:], CV_CTX + H(ch + sh + ee + cert).digest())
    if not hmac.compare_digest(sfin[4:], finished(derive(hs_c, b"s hs traffic", ch + sh),
                                                  ch + sh + ee + cert + cv)):
        raise ValueError("server Finished mismatch")
    cfin = msg(20, finished(c_hs, ch + sh + ee + cert + cv + sfin))
    # server checks the client Finished
    if not hmac.compare_digest(cfin[4:], finished(c_hs_srv, ch + sh + ee + cert + cv + sfin)):
        raise ValueError("client Finished mismatch")
    master = extract(derive(hs_c, b"derived", b""), b"\0" * HLEN)
    return derive(master, b"c ap traffic", ch + sh + ee + cert + cv + sfin)

key = Ed25519PrivateKey.generate()
print("client app secret", handshake(key, raw(key.public_key())).hex()[:16])
handshake(key, raw(key.public_key()), tamper=True)   # raises InvalidSignature

Worked example: tampering and downgrade

Run the toy and it prints a client application secret, different on every run because both key shares are fresh. Now walk through the tampered run. The attacker rewrites two bytes of ServerHello after the server has computed its signature. The client hashes the ServerHello it received, so its transcript hash differs from the one the server signed, and Ed25519 verification raises InvalidSignature. Remove the signature check and the attack still fails one step later: the server Finished MAC was computed over the server's transcript, the client recomputes it over its own, and the comparison fails. Remove that too and the client and server would derive different application keys, so the first record would fail to decrypt. Defence in depth is built into the schedule itself.

Now consider a real downgrade attempt. An attacker strips TLS 1.3 from supported_versions in the ClientHello so the server negotiates TLS 1.2. The server, which supports 1.3, writes the downgrade sentinel into its random. The client sees 1.2 negotiated, checks the last eight bytes of the server random, finds the sentinel, and aborts. Because the random is also signed, the attacker cannot remove the sentinel without breaking the signature.

Resumption and early data

After the handshake the server can send a NewSessionTicket. On a later connection the client offers that ticket as a pre-shared key, and the state machine takes the short path: no Certificate or CertificateVerify, because the PSK itself proves the earlier authentication. With the psk_dhe_ke mode the client still sends a key share, so the new session keeps forward secrecy. A PSK also enables 0-RTT early data encrypted under a key derived from the early secret, but that data can be replayed by an attacker, so it must be limited to idempotent requests. The operational side of tickets and 0-RTT is covered in TLS handshake optimization.

Failure modes

• Skipping states. Accepting Finished without having seen CertificateVerify. Use an explicit transition table and a fatal default.
• Hostname not checked. Validating the chain but not that it names the host you dialled. The handshake then authenticates someone, just not your server.
• Non-constant-time comparison. Comparing MACs with == leaks timing. Use hmac.compare_digest or the library equivalent.
• Unvalidated key shares. Accepting invalid curve points or the all-zero X25519 output. Libraries handle this; hand-rolled code often does not.
• 0-RTT on non-idempotent endpoints. Replayed early data charges a card twice. Gate early data at the application.
• Middlebox interference. Old inspection boxes drop unfamiliar messages, which is why TLS 1.3 has a compatibility mode that makes it look like a TLS 1.2 resumption on the wire.

Trade-offs

TLS 1.3 traded flexibility for safety. Removing static RSA, renegotiation and many cipher suites shrank the state machine and closed whole classes of attack, at the cost of breaking passive decryption tools that relied on a static server key; operators now export session keys instead. Guessing the key share saves a round trip but wastes one when the guess is wrong, which is why clients usually send a share for their most likely group, and some send two. Encrypting the certificate hides which site a client visits from passive observers but makes network debugging harder. For service-to-service traffic where both sides present certificates, see mTLS for service-to-service traffic.

What to do next

1. Run the toy, then delete each verification step in turn and predict which later check catches the tampering.
2. Capture a real handshake with openssl s_client -connect example.com:443 -tls1_3 -msg and match each message to the table above.
3. Set SSLKEYLOGFILE for a browser or curl and decrypt the handshake in Wireshark to see the encrypted server flight.
4. Audit your own client code: check that hostname verification is on and that no option disables certificate checks.
5. Confirm your servers support X25519 so that common clients avoid a HelloRetryRequest.
6. Decide whether any endpoint needs 0-RTT, and if so restrict it to idempotent requests.
7. Read RFC 8446 sections 4 and 7 alongside the internals page to connect messages to bytes.