# Signal Protocol Reference This document explains how @mataram/wa implements the Signal end-to-end encryption protocol for WhatsApp messaging. --- ## 1. What the Signal Protocol Is The Signal Protocol provides **end-to-end encryption** (E2EE) — messages are encrypted on the sender's device and can only be decrypted on the intended recipient's device. WhatsApp uses the protocol for all private messaging. @mataram/wa wraps the [`libsignal`](https://www.npmjs.com/package/libsignal) library (version 6.x) to implement: - **X3DH** (Extended Triple Diffie-Hellman) for session establishment - **Double Ratchet** for per-message key rotation - **Sender Keys** for efficient group encryption All cryptographic material is stored through the `SignalKeyStore` interface, allowing any storage backend (JSON file, SQLite, Redis, etc.). ```ts const makeSignalRepository = ({ creds, keys }, logger, pnToLIDFunc) ``` The repository (`Signal/libsignal.ts`) connects `libsignal`'s storage callbacks to the library's key store, handling JID resolution, session migration, and atomic transactions. --- ## 2. Key Types ### Identity Key (`signedIdentityKey`) - **Type**: Curve25519 key pair (long-term, permanent) - **Storage**: In-memory in `AuthenticationCreds.signedIdentityKey` - **Purpose**: Establishes the long-term identity of the device. The public half is uploaded to the server during registration and included in the account signature. - **Lifetime**: Never changes for the lifetime of the credential file ### Signed Pre-Key (`signedPreKey`) - **Type**: Curve25519 key pair + signature (medium-term) - **Storage**: `AuthenticationCreds.signedPreKey` - **Purpose**: Used during X3DH session establishment. The private key is signed with the identity key so recipients can verify authenticity. - **Rotation**: The server can request rotation via `rotateSignedPreKey()`. The key ID increments monotonically. ### One-Time Pre-Keys (`pre-key`) - **Type**: Curve25519 key pairs (ephemeral, single-use) - **Storage**: `SignalKeyStore`, keyed as `'pre-key'` with numeric string IDs - **Purpose**: Pre-keys are uploaded to the server in batches (100-500). Each one is consumed once when someone establishes a session with you. Having a pool of pre-keys ensures asynchronous session establishment — Alice can encrypt a message for Bob even if Bob is offline. - **Tracking**: `creds.nextPreKeyId` and `creds.firstUnuploadedPreKeyId` track the range of generated keys. ### Ephemeral Key - **Type**: Curve25519 key pair (per-session) - **Storage**: Generated fresh per connection in `socket.ts` for the Noise handshake; within X3DH, the sender generates a per-session ephemeral - **Purpose**: Provides forward secrecy — if the long-term keys are compromised, past session keys remain secure ### Chain Key & Message Key - **Storage**: Inside the Signal session record (serialized as `'session'`) - **Chain Key (CK)**: Derives message keys via a one-way ratchet function. Each message advances the chain. - **Message Key (MK)**: Actual AES-256-GCM key used to encrypt/decrypt a single message. Derived from the chain key. --- ## 3. Session Establishment (X3DH) When Alice first messages Bob, the library performs X3DH key agreement: ### Initiator (Alice) side 1. **Fetch pre-key bundle**: The library sends an IQ to the server requesting Bob's key bundle: ``` ``` 2. **Parse response**: `parseAndInjectE2ESessions()` extracts Bob's identity key, signed pre-key, and (optionally) one-time pre-key from the server response. 3. **Inject session**: `repository.injectE2ESession()` creates a new Signal session by calling `SessionBuilder.initOutgoing()` with the bundle. 4. **Encrypt**: The first message is encrypted as a **PreKeyWhisperMessage** (`type: 'pkmsg'`), which contains the sender's identity key, ephemeral key, and the encrypted message payload. ### Recipient (Bob) side 1. **Receive PKMSG**: The library detects a `pkmsg` type in the `` node. 2. **Decrypt**: `session.decryptPreKeyWhisperMessage()` uses Bob's stored pre-key, signed pre-key, and identity key to compute the shared secret and decrypt the message. 3. **Consume pre-key**: The one-time pre-key is removed from storage. 4. **Establish session**: Both sides now have a shared root key, from which all subsequent message keys are derived via the Double Ratchet. ### Session info ```ts const info = await signalRepository.getSessionInfo(jid) // { baseKey: Uint8Array, registrationId: number } ``` --- ## 4. Session Lifecycle ### Creating Sessions are created automatically when needed: - **Outgoing**: `assertSessions()` checks if a session exists; if not, fetches the recipient's pre-key bundle and calls `injectE2ESession()`. - **Incoming**: Receiving a `pkmsg` (pre-key message) automatically creates a session on the recipient side. ### Storing Sessions are stored in the `'session'` key type of `SignalKeyStore`. The storage key is the output of `jidToSignalProtocolAddress()`: ```ts function jidToSignalProtocolAddress(jid: string): string { const decoded = jidDecode(jid) const signalUser = domainType !== WHATSAPP ? `${user}_${domainType}` : user return `${signalUser}.${device || 0}` } ``` ### Loading `loadSession()` in the storage adapter: 1. Resolves the JID to a wire address (handling PN→LID mapping) 2. Calls `keys.get('session', [wireAddress])` 3. Deserializes with `libsignal.SessionRecord.deserialize()` ### Migrating (PN → LID) See [§8 LID Mapping](#8-lid-mapping). --- ## 5. Sender Keys (Group Encryption) Groups use **sender key encryption** — one ciphertext for all group members, dramatically reducing the overhead of per-device encryption. ### How it works 1. **First message**: The sender creates a `SenderKeyRecord` and distributes it via a **SenderKeyDistributionMessage** (SKDM) encrypted to each recipient's individual session (as `pkmsg`/`msg`). 2. **Subsequent messages**: The sender encrypts using the symmetric ratchet from the sender key. All recipients use the same sender key to decrypt. 3. **SKDM re-distribution**: When new members join, or when existing members don't have the sender key (tracked via `'sender-key-memory'`), the SKDM is re-sent. ### Implementation ```ts // Encryption (per group message) const { ciphertext, senderKeyDistributionMessage } = await signalRepository.encryptGroupMessage({ group: '123@g.us', data: bytes, meId: 'sender@lid' // identity used for the sender key name }) // Decryption (per group message) const plaintext = await signalRepository.decryptGroupMessage({ group: '123@g.us', authorJid: 'sender@lid', msg: ciphertext }) ``` ### Sender key name format The sender key is identified by `SenderKeyName(group, senderAddress)`: - `group` = the group JID - `senderAddress` = the sender's ProtocolAddress (user + device) This ensures each sender has their own sender key chain within a group. --- ## 6. Message Encryption (Double Ratchet) The Double Ratchet algorithm provides **forward secrecy** and **future secrecy** (also called post-compromise security). ### Forward secrecy Each message key is derived from the previous chain key. Even if the current chain key is compromised, past message keys cannot be reconstructed. ### Self-healing (future secrecy) The **DH ratchet** (the "double" part) introduces new ephemeral key exchanges periodically. When Alice receives a message with a new DH public key, she ratchets her sending and receiving chains. This means if an attacker compromises the current state, they can only decrypt messages until the next DH ratchet step. ### Encryption flow ```ts const { type, ciphertext } = await signalRepository.encryptMessage({ jid: 'recipient@s.whatsapp.net', data: encodedMessageBytes }) ``` - `type` is `'pkmsg'` if this is the first message (or after session loss), `'msg'` otherwise. - `ciphertext` is the AES-256-GCM encrypted payload. ### Decryption flow ```ts const plaintext = await signalRepository.decryptMessage({ jid: senderJid, type: 'pkmsg' | 'msg', ciphertext }) ``` The library wraps each decrypt call in a transaction to ensure atomicity of session state updates. --- ## 7. Pre-Key Management ### Generation Pre-keys are generated on-demand in `getNextPreKeys()` (`Utils/signal.ts`): ```ts const { newPreKeys, lastPreKeyId, preKeysRange } = generateOrGetPreKeys(creds, count) // Count is typically 20 (MIN_PREKEY_COUNT) or 100 (INITIAL_PREKEY_COUNT) ``` Each generated key pair is immediately stored to the key store to prevent ID collisions on retry: ```ts const node = await keys.transaction(async () => { const { update, node } = await getNextPreKeysNode({ creds, keys }, count) ev.emit('creds.update', update) // persist new nextPreKeyId return node }, 'upload-pre-keys') ``` ### Upload The upload flow: 1. `uploadPreKeysToServerIfRequired()` checks server pre-key count via `` 2. If low (≤ `MIN_PREKEY_COUNT`) or the current pre-key is missing, `uploadPreKeys()` is called. 3. The upload node includes the registration ID, key bundle type, identity key, a list of pre-keys, and the signed pre-key: ``` ... 05 ... 001... 002... 001...... ``` ### Low pre-key notifications The server sends `CB:notification,type:encrypt` when pre-keys run low. The handler checks the `` value and triggers re-upload if it's below the minimum threshold. The same stanza ID is tracked via `inFlightPreKeyLow` to skip duplicate notifications. ### PreKeyManager The `PreKeyManager` (`Utils/pre-key-manager.ts`) provides concurrency control for pre-key operations: ```ts class PreKeyManager { private readonly queues = new Map() async processOperations(data, keyType, transactionCache, mutations, isInTransaction) async validateDeletions(data, keyType) } ``` It uses per-key-type queues (`PQueue` with concurrency 1) to serialize operations, and validates deletions against the store to prevent accidental data loss. --- ## 8. LID Mapping WhatsApp assigns each user a **LID** (Long-term ID) — a stable internal identifier that never changes, even if the user changes their phone number. Phone numbers (PNs) can change, but the LID remains constant. ### Why migration is needed When WhatsApp migrated users from PN-based to LID-based addressing, existing Signal sessions were keyed by phone number (`user@s.whatsapp.net`). The LID system requires sessions to be keyed by the user's LID (`user_lid@lid`). Sessions must be **copied** from the PN address to the LID address. ### LIDMappingStore The `LIDMappingStore` (`Signal/lid-mapping.ts`) manages PN↔LID mappings: ```ts class LIDMappingStore { // Caches mappings for 7 days private mappingCache = new LRUCache() async storeLIDPNMappings(pairs: LIDMapping[]): Promise async getLIDForPN(pn: string): Promise async getLIDsForPNs(pns: string[]): Promise async getPNForLID(lid: string): Promise async getPNsForLIDs(lids: string[]): Promise } ``` Mappings are stored in the `'lid-mapping'` key store and cached in an LRU cache. Lookups are coalesced — duplicate in-flight requests share a single promise. ### Session migration When a PN→LID mapping is discovered (via history sync, message envelope, USync, or LID migration protocol message), the library migrates all device sessions: ```ts await signalRepository.migrateSession('user@s.whatsapp.net', 'user_lid@lid') ``` The migration process: 1. Loads the user's device list from `'device-list'` storage 2. For each device that has a PN-keyed session, serializes the session record and writes it to the LID address 3. Deletes the PN-keyed session 4. Caches the migration result per device to avoid re-migration ### Source of mappings | Source | Trigger | |---|---| | History sync | `LID_PN_MAPPINGS` in history sync notification | | Message envelopes | `participant_lid`/`participant_pn` attributes on received stanzas | | USync | Device discovery queries with LID protocol | | `LID_MIGRATION_MAPPING_SYNC` | Protocol message from own phone | | Linked profiles notifications | Mex notification `NotificationLinkedProfilesUpdates` | --- ## 9. Error Recovery ### NoSession When decrypting a message and no session exists for the sender, the message decryption fails. The library: 1. Sends a NACK with a retry request, including the recipient's registration ID and (on retry count > 1) fresh pre-keys 2. The sender re-sends the message, this time including a pre-key bundle so a new session can be established 3. The recipient injects the session from the retry receipt's key bundle via `injectE2ESession()` ### BadMAC A bad MAC (Message Authentication Code) indicates the session is out of sync — the recipient has ratcheted to a different state than the sender. The `MessageRetryManager` detects this via the error code in the retry receipt: ```ts enum RetryReason { SignalErrorNoSession = 1, SignalErrorInvalidKey = 2, SignalErrorInvalidKeyId = 3, SignalErrorInvalidMessage = 4, SignalErrorInvalidSignature = 5, SignalErrorFutureMessage = 6, SignalErrorBadMac = 7, // Explicit MAC failure SignalErrorInvalidSession = 8, SignalErrorInvalidMsgKey = 9, // ... } ``` When a MAC error (codes 4 or 7) is detected: 1. The session is immediately deleted 2. The retry forces `forceIncludeKeys = true`, so fresh pre-keys are sent with the retry receipt 3. The sender establishes a new session ### InvalidKeyId A stale pre-key was referenced. The library re-fetches the full key bundle from the server and injects a fresh session. ### Base key collision detection The `MessageRetryManager` tracks base keys per session: ```ts // At retry count 2, save the base key messageRetryManager.saveBaseKey(sessionId, msgId, info.baseKey) // At retry count > 2, compare against stored base key if (messageRetryManager.hasSameBaseKey(sessionId, msgId, info.baseKey)) { // Same base key after multiple retries → force fresh session await authState.keys.set({ session: { [sessionId]: null } }) } ``` This catches cases where session recreation isn't working and the same broken session is being re-created. ### Retry mechanism The retry system works in layers: 1. **Incoming retry**: If the sender can't decrypt a message, they send a `` back to the sender. The original sender looks up the message (from cache or `getMessage()`) and re-encrypts it under a fresh session. 2. **Phone request**: On the first two retries, a `PeerDataOperationRequestMessage` is sent to the user's phone requesting a placeholder resend. The phone may respond with the full message bytes. 3. **Session recreation**: If `enableAutoSessionRecreation` is true and retries exceed 1, the `MessageRetryManager` decides whether to delete and re-establish the session based on: - Whether a session exists at all - The error code (MAC errors trigger immediate recreation) - Time since last recreation (at most once per hour) 4. **Retry codes 0-13**: Map to Signal error types via `RetryReason` enum. Retry receipts include an `error` attribute with the code, enabling targeted recovery strategies. --- ## 10. Privacy Tokens (tctoken) Privacy tokens (`tctoken`) are short-lived cryptographic tokens that WhatsApp uses to gate 1:1 messaging to accounts with an established relationship. ### What they are A tctoken is an opaque buffer issued by WhatsApp's privacy token service (`xmlns='privacy'`). Each 1:1 conversation partner has one stored under the `'tctoken'` key type in `SignalKeyStore`. ### Issuance Tokens are issued via an IQ: ```ts const result = await query({ tag: 'iq', attrs: { to: S_WHATSAPP_NET, type: 'set', xmlns: 'privacy' }, content: [ { tag: 'tokens', attrs: {}, content: [{ tag: 'token', attrs: { jid: contactJid, t: timestamp, type: 'trusted_contact' } }]} ] }) ``` ### Storage Tokens are stored under a JID-based key: ```ts { 'tctoken': { 'user_lid@lid': { token: Buffer, // the opaque token timestamp: '1234567890', // issuance timestamp senderTimestamp: 1234567890 // when we last sent a token to this contact } } } ``` ### Sending Before sending a 1:1 message, the library checks for a non-expired tctoken and attaches it as a `` child node: ```ts if (tcTokenBuffer?.length && sock.serverProps.privacyTokenOn1to1) { stanza.content.push({ tag: 'tctoken', attrs: {}, content: tcTokenBuffer }) } ``` ### Expiry Tokens expire after ~28 days (4 buckets × 7 days). The `isTcTokenExpired()` function checks if the timestamp falls within the current rolling window. ### Error handling If a message is rejected with server error `'463'` (MessageAccountRestriction), it means the account is restricted from starting new conversations — the `tctoken` is missing. --- ## 11. SignalRepository Interface The `SignalRepository` (`Types/Signal.ts`) is the main interface for all cryptographic operations: ```ts interface SignalRepository { decryptMessage(opts: { jid: string; type: 'pkmsg' | 'msg'; ciphertext: Uint8Array }): Promise encryptMessage(opts: { jid: string; data: Uint8Array }): Promise<{ type: 'pkmsg' | 'msg' ciphertext: Uint8Array }> encryptGroupMessage(opts: { group: string; data: Uint8Array; meId: string }): Promise<{ senderKeyDistributionMessage: Uint8Array ciphertext: Uint8Array }> decryptGroupMessage(opts: { group: string; authorJid: string; msg: Uint8Array }): Promise getSenderKeyDistributionMessage(opts: { group: string; meId: string }): Promise hasSenderKey(opts: { group: string; meId: string }): Promise processSenderKeyDistributionMessage(opts: { item: proto.Message.ISenderKeyDistributionMessage authorJid: string }): Promise getSessionInfo(jid: string): Promise<{ baseKey: Uint8Array registrationId: number } | null> injectE2ESession(opts: { jid: string; session: E2ESession }): Promise validateSession(jid: string): Promise<{ exists: boolean; reason?: string }> jidToSignalProtocolAddress(jid: string): string migrateSession( fromJid: string, toJid: string ): Promise<{ migrated: number; skipped: number; total: number }> deleteSession(jids: string[]): Promise } ``` The `SignalRepositoryWithLIDStore` extension adds: ```ts interface SignalRepositoryWithLIDStore extends SignalRepository { lidMapping: LIDMappingStore close?: () => void } ``` This is what `makeLibSignalRepository()` in `Signal/libsignal.ts` creates. The implementation: - Wraps all cryptographic calls with `parsedKeys.transaction()` for atomicity - Provides LID resolution in `loadSession` and `storeSession` via `resolveLIDSignalAddress()` - Uses TOFU (Trust On First Use) for identity key verification - Caches migrated sessions to avoid repeated migration work - Returns identity change notifications from `saveIdentity()` for upstream handling (identity change events, tctoken re-issuance)