How Group Encrypted Messaging Actually Works

Source: Dev.to

Background

Adding a third person to an encrypted conversation seems like it should be simple, but it isn’t. The cryptographic properties that make 1:1 messaging secure—forward secrecy, post‑compromise security, deniability—become significantly harder to preserve as group size grows.

When Signal introduced group chats, they faced a problem that doesn’t exist in 1:1 messaging: how do you efficiently encrypt a single message for many recipients while preserving strong security guarantees? The naive answer—encrypt the message separately for each recipient—works but scales poorly.

Pairwise Encryption (Signal v1)

The simplest group‑messaging implementation is pairwise encryption: the sender establishes a separate secure channel with each group member and sends the message individually to each one. This provides full forward secrecy because each channel uses the Signal Protocol (Double Ratchet), which rotates keys on every message.

• Pros: Strong security; each pairwise channel is as secure as a 1:1 conversation.
• Cons: Linear scaling—message overhead grows proportionally with group size. For a group of 100 people, sending one message requires 100 individual encryptions, 100 network deliveries, and 100 ratchet advances.

Sender Keys (Signal current)

Signal’s current group implementation uses Sender Keys to solve the scaling problem.

1. Each group member generates a Sender Key—a chain key used to derive a sequence of encryption keys for that member’s outgoing messages.
2. The Sender Key is distributed to each group member individually (one‑time cost).
3. After that, the sender encrypts each message once with the current key from their chain.

This reduces message overhead from O(N) to O(1) for subsequent messages.

• Trade‑off: Sender Keys advance forward through a hash chain, providing forward secrecy against passive observers, but they lack the full Double Ratchet’s break‑in recovery. If a member’s current state is compromised, an attacker can decrypt subsequent messages until the chain is reset.
• Result: Forward secrecy is retained, but post‑compromise security is weaker.

Membership Changes

The hardest problem in secure group messaging isn’t encrypting messages—it’s managing membership changes securely.

• Forward secrecy for new members: New members must not decrypt messages sent before they joined.
• Post‑remove secrecy: Removed members must not decrypt future messages. Achieving this requires rotating the group key after removal, which for a group of 1,000 members means 999 individual key distributions under pairwise or Sender‑Key schemes.

Messaging Layer Security (MLS, RFC 9420)

Messaging Layer Security (MLS) was designed from first principles to solve group‑messaging security at scale.

• MLS replaces the O(N) key distribution on membership change with a binary tree (the “ratchet tree”) where group members occupy the leaves and internal nodes hold intermediate keys.
• When a member updates their key material (a “commit”), only the nodes on the path from their leaf to the root need updating. In a balanced tree with N members, that’s O(log N) operations rather than O(N)—roughly 10 operations for 1,000 members instead of 999.

Epochs

MLS introduces the concept of an epoch—a version of the group state. Every time the group membership changes or key material is updated, the group advances to a new epoch. Messages are encrypted under the epoch key valid at the time of sending. This provides:

• Clean boundaries for forward secrecy.
• Recovery from device compromise by rotating to a new epoch.
• A cryptographic audit trail of group‑state changes.

Concurrency

In a distributed system, two members might try to advance the epoch simultaneously—a “concurrent commit.” MLS defines a resolution mechanism, adding complexity that Sender‑Key approaches avoid.

Adoption (as of 2026)

• WhatsApp announced MLS adoption.
• Apple Messages began using MLS for group chats.
• Matrix has an MLS implementation in development.
• Haven uses MLS for encrypted group chat, specifically for the O(log N) membership‑change overhead and strong post‑compromise security guarantees.
• Signal continues using Sender Keys for large groups—both positions are defensible depending on group‑size distributions and adversary models.

Comparison Table

Key Questions for Secure Group Messaging Apps

1. Does removing a member prevent them from reading future messages?
2. Can a new member read history from before they joined?
3. Is the group protocol documented and independently audited?
4. What happens if one member’s device is compromised?

Forward secrecy in 1:1 contexts is well‑understood; in group contexts it requires explicit design choices that vary significantly between implementations.

Originally published at havenmessenger.com.