What Is a VPN Protocol?

A VPN protocol is a specification for how a tunnel is built: what goes into it, how it is encrypted, how each end proves who it is, and what the bytes look like on the wire. Choosing one rests on more than speed, because the wire appearance is what a censor sees.

Short answer#

A VPN protocol has to answer four separate questions:

1. Transport: what carries the tunnel? UDP, TCP, or something shaped like web traffic
2. Authentication: how do the two ends recognise each other?
3. Encryption: what protects the payload?
4. Framing: what does the encrypted stream look like from the outside?

Most protocols answer the first three competently. The fourth is where they differ from each other, and it is the only one a censor can act on without breaking the cryptography.

The families#

Layer-3 IP tunnels#

The tunnel carries IP packets, and your operating system routes into it as if it were a network interface. This is the traditional definition of a VPN.

IPsec with IKEv2 is the standardised one, built into operating systems, and designed for site-to-site networks as much as for individuals.

WireGuard takes a modern line on the same idea: a layer-3 tunnel over UDP, with a deliberately small codebase and a fixed set of cryptographic primitives.

OpenVPN sits between these two families, because it runs its layer-2 or layer-3 tunnel inside TLS. Its own documentation describes it as a full-featured SSL VPN implementing OSI layer 2 or 3 extension using the standard TLS protocol. That flexibility lets it run over TCP or UDP, and through NAT, and it also fixes its outer layer as TLS, which is the most closely inspected protocol on the internet.

Proxy protocols#

A proxy protocol is not a network tunnel but a forwarding format. Your application hands it a destination, and it opens the connection.

Shadowsocks is an encrypted SOCKS proxy. It was designed to be simple and unremarkable, and it became one of the most widely used circumvention tools in the world, which also made it one of the most studied. Research presented at ACM IMC in 2020 documented exactly how a national filtering system identified it.

VLESS is a much thinner protocol, described by its own documentation as a stateless lightweight transport that bridges client and server. It delegates all encryption to whatever transport carries it.

Disguised transports#

The third family does not replace a protocol; it wraps one in something already common on the internet. This is where the current design work sits, because it is the only family whose answer to question four is “make me look like something else”.

How they compare#

The last two rows are the ones that matter on a restricted network, and they are the reason the newest family exists.

Why the protocol is a censorship question#

Everything in the previous section is invisible to a censor, because all of it happens inside the tunnel. The one thing that is not hidden is the tunnel itself.

This is not a theoretical concern. A 2024 paper in Communications of the ACM demonstrated that OpenVPN traffic is statistically distinguishable and can be classified. The general finding is worse than the specific one: encrypted VPN traffic carries a signature in its packet sizes and timing that a classifier can learn.

So the protocol choices divide cleanly into two questions:

• If nobody is trying to block you, pick on operational grounds. WireGuard is small and fast; OpenVPN is flexible and ubiquitous; IKEv2 needs no extra software.
• If somebody is trying to block you, the wire appearance decides, and that is where VLESS with REALITY is a different proposition from the rest. The project’s own documentation is explicit that REALITY removes server-side TLS fingerprint characteristics while keeping forward secrecy.

Neither question has a single right answer, which is why both exist.

Choosing#

Some rules that hold regardless of what else you care about:

• Prefer a protocol with a small surface. Fewer options means fewer distinguishable behaviours. WireGuard’s design premise is that you cannot extend it, which is the point.
• Check the transport, not just the protocol. A good protocol on a bad transport inherits the transport’s problems.
• Treat “undetectable” as unproven. Any protocol whose documentation promises invisibility is making a claim about an adversary’s capabilities. Ask what was measured, against what, and by whom. Atlas treats such claims as claims.
• Prefer boring to clever, unless you have a reason. The oldest advice in this field remains correct for most people most of the time.

Common misconceptions#

“The protocol decides the security.” It decides part of it. Client authentication, endpoint configuration, patching, and who can reach the server all matter as much, and none of them are in the specification.

“OpenVPN is insecure.” The claim in the literature is specific: that its traffic is distinguishable, not that its cryptography is broken. Fingerprintability and cryptographic weakness are different problems with different fixes.

“UDP is faster than TCP.” Usually, and not always. UDP has no congestion control, so a WireGuard connection over a lossy path can be worse than a TCP-based tunnel that adapts.

“Any protocol will work if I wrap it in TLS.” Wrapping one encrypted protocol inside another produces a recognisable structure, and the Xray project warns about exactly this: combining REALITY with other proxy protocols produces clear TLS-in-TLS characteristics that have already been targeted.

“Newer is better.” WireGuard is newer than OpenVPN and much simpler; that is a real advantage in some ways and a real limitation in others, since it deliberately omits anything that would allow the protocol to grow.

Related#

• What Is WireGuard? — the smallest, most opinionated design
• What Is VLESS? — the thinnest protocol in the set
• What Is REALITY? — the transport layer that answers question four
• What Is Deep Packet Inspection? — why the wire appearance matters
• What Is Hysteria2? — a QUIC proxy that bets on loss and throughput instead of camouflage

Last reviewed October 30, 2026. This article was revised after publication. Corrections and additions are welcome.