The Story of How I Built a VPN protocol: Part 1

Source: Dev.to

Disclaimer

This article and the VPN itself are written for educational purposes only.

How It All Started

I recently switched to Arch. Everything started off well: I installed all the utilities I needed, and then I decided to install the VPN I used to use. A problem appeared — it doesn’t work on Arch (even as an AppImage).

My provider also supported Shadowsocks, but instead of using it I decided to write my own VPN for more practice.

VPN Protocol

My VPN protocol is designed for maximum stealth. In my opinion, one of the most important things here is encryption from the very first packet. In my protocol this is implemented just like in Shadowsocks — with a pre‑shared key.

• Encryption algorithm: ChaCha20‑Poly1305
• Transport: TCP (a random amount of junk bytes is added to each packet for length obfuscation)

Packet Structure

Each packet has a 5‑byte header that is masked as encrypted data using XOR with the first 5 bytes of the key.

After the header the packet contains:

1. 12 bytes – randomly generated nonce
2. ciphertext
3. AEAD (authentication tag)
4. junk bytes

Handshake & Key Exchange

1. First packet from the client

The client sends its 16‑byte username to the server (encrypted, of course).

2. Server response

If the server finds a user with that username, it:

• Sends the client a randomly generated 32‑byte salt
• Starts computing the keys:
  • Sending key (server → client)
  • Receiving key (server ← client)

3. Key computation on the server

The server stores the user’s password in plaintext.

• Receiving key (for decrypting from the client) = hash(password + first 16 bytes of salt)
• Sending key (for encrypting to the client) = hash(password + last 16 bytes of salt)

4. Client actions

The client receives the salt, decrypts it, and does the same thing, but the key roles are inverted:

• What is the sending key for the server becomes the receiving key for the client, and vice‑versa.

5. ECDH and connection finalisation

1. After the client has generated the keys, it creates an ephemeral key pair based on the Curve25519 elliptic curve (needed for ECDH).
2. It then sends a connection confirmation (0xFF) together with its public ephemeral key, setting the ECDH flag.

The server receives the packet, de‑obfuscates it, and obtains the confirmation and the client’s ephemeral key. It then:

• Assigns an IP address to the client from a local private network
• Generates its own ephemeral key pair
• Sends the client its assigned IP address and the server’s public key
• Performs the ECDH round

After sending, the server updates its keys by hashing the old keys with the secret obtained from ECDH.

6. Client finalisation

When the client receives the packet containing the IP address and the server’s public ephemeral key, it:

• Creates a local tunnel
• Sets its IP address (the one received from the server)
• Performs the ECDH round
• Updates its keys

Main Work Loop

After the connection is established and keys are generated, the main work loop begins.

Client Side

Three goroutines run on the client side:

1. Reading from the tunnel & preparing packets

   • Reads packets from the tunnel.
   • Generates an 8‑byte salt to update the sending key (by hashing the old sending key with the salt).
   • Prepends this salt to the plaintext (salt + tunnel packet).
   • Encrypts everything.
   • Adds random junk bytes for obfuscation.
   • Stores the prepared packet in a buffer.

2. Sending packets

   • Sends the already‑prepared packets.
   • Packets are sent in batches of 1‑5 packets (the protocol is, of course, segmented at OSI layers 3 and 4, but I can’t influence that).

3. Receiving packets from the server

   • Receives packets from the server.
   • Performs de‑obfuscation and decryption.
   • Writes the decrypted data to the tunnel.

Server Side

The server has three main goroutines, plus additional goroutines for receiving packets from clients.

1. Handshake handling – processes incoming handshake requests. If the handshake succeeds, a new goroutine is spawned to process packets from that client.
2. Reading from the tunnel – reads packets from the tunnel and forwards them to clients.
3. Cleaning inactive connections – removes stale connections.

Key Updates

Salt in every packet

Every packet (whether from client or server) contains a salt used to update the keys:

• Server (sending): includes a salt, then updates its sending key by hashing the old key with that salt.
• Client (receiving): after decrypting a packet, updates its receiving key with the same salt.

When the client sends a packet, the roles are reversed (client updates its sending key, server updates its receiving key).

Periodic ECDH updates

Every 4 minutes or after sending 2³² packets (whichever comes first), keys are refreshed using an ECDH exchange on elliptic curves. The new keys are transmitted together with data packets.

Implementation

During implementation I considered writing it in Go or Rust. I chose Go for its simplicity.

End of protocol description.

On Process

To be honest, the protocol architecture was mostly developed while writing the code. It has quite a few problems — both in terms of protocol design and implementation.

Example problems

• Constant username packet length
  The encrypted username packet has a constant length of 44 bytes (12 bytes nonce, 16 bytes ciphertext, and a 16‑byte AEAD tag). Knowing this and that the user is using this protocol, you can calculate the 4th and 5th bytes of the key.

• Repository duplication
  I foolishly created two separate repositories — one for the client and one for the server. As a result, the branches containing common modules just duplicate each other.

• Git flow
  I tried to follow Git Flow, but failed here too.

• Vulnerabilities
  I also have a feeling that there are more vulnerabilities in the code than working logic.

• No graceful shutdown
  There is no proper negotiated client‑server disconnect — just a connection break.

Although this is my first project, I think it didn’t turn out too badly. If anyone wants to check out this mess, here are the links:

• Client:
• Server:

The implementation currently works, and I’m writing this article through my own VPN protocol.

Future Plans

• Merge both repositories into one.
• Add fake packet sending.
• Add TLS mimicry.
• And much more.

If anyone has any questions or recommendations, leave them in the comments. For now, I bid you farewell. Good luck to everyone!