ARC4 Decryption Succeeding on SMSG_AUTH_RESPONSE but Failing on SMSG_CHAR_ENUM

ARC4 decryption fails on SMSG_CHAR_ENUM because the packet length header is consumed during parsing but not excluded from the ciphertext passed to the cipher. This causes a byte desynchronization where the stream cipher starts at an offset one byte too late, corrupting the character list while leaving earlier packets like SMSG_AUTH_RESPONSE intact.

The Root Cause: Header Consumption vs. Ciphertext Offset

In AzerothCore and World of Warcraft private server emulation, network packets follow a specific binary structure. The standard format for most client-to-server or server-to-client messages includes a two-byte opcode followed by a variable-length payload. However, the critical detail often missed in custom Python clients is how the length is handled relative to the encryption boundary. When you receive data from the socket, it typically arrives as a raw byte stream. For a packet like SMSG_AUTH_RESPONSE, the structure might look like this:

1. Opcode (2 bytes): Identifies the message type.
2. Length (2 bytes): Specifies the size of the encrypted payload that follows.
3. Ciphertext (N bytes): The actual encrypted data.

The error occurs when your code reads the first four bytes (opcode + length) to determine what kind of packet it is. If you then pass the remaining buffer directly to the ARC4 (RC4) decryption function without accounting for the fact that the length field itself is unencrypted, you are effectively skipping the correct number of bytes only if the length field is part of the plaintext. In WoW protocol, the length field is usually sent in plaintext, but the *encryption* starts immediately after the length field. If your decryption routine consumes the length bytes but the cipher state was initialized based on the full packet including those bytes, or if you skip them incorrectly, the RC4 keystream becomes misaligned. Since SMSG_AUTH_RESPONSE is small and simple, minor offsets might not cause total failure, but SMSG_CHAR_ENUM contains a large array of character structures. Even a single-byte shift in the RC4 stream will result in garbled output for the entire remainder of the packet, causing the parser to fail completely.

Correcting the Decryption Logic

To fix this, you must ensure that the RC4 cipher is initialized with the key derived from the session, and that the input to the decrypt method consists *only* of the bytes between the end of the length header and the end of the packet. Here is the correct approach using pycryptodome:

from Crypto.Cipher import ARC4
import struct

def decrypt_wow_packet(key, raw_data):
    # 1. Parse the header
    # Assuming little-endian format common in WoW
    opcode = struct.unpack_from('

The critical line here is ciphertext = raw_data[start_offset:start_offset + length]. Many developers mistakenly slice raw_data[4:] assuming the rest of the buffer is always the payload. However, if there is trailing garbage or if the packet is fragmented, relying on the explicit length field is safer. More importantly, verify that the key used for SMSG_AUTH_RESPONSE is identical to the one used for SMSG_CHAR_ENUM. If the session key changes mid-stream due to a re-keying event, you must update the cipher instance.

Debugging Byte Alignment

If the above logic still results in corruption, you need to verify the byte alignment visually. Print the first 16 bytes of the decrypted plaintext for both packets.

For SMSG_AUTH_RESPONSE, you should see valid structure headers or known constants. For SMSG_CHAR_ENUM, you expect to see a series of character records. Each character record in AzerothCore typically starts with a uint32 GUID. If your decrypted output shows random gibberish instead of sequential integers or recognizable ASCII strings for names, your keystream is off.

You can also test the hypothesis by manually shifting the decryption start point. Try decrypting with start_offset = 5 or start_offset = 3. If start_offset = 3 yields readable text, it means the length field was actually 1 byte in your specific build, or the opcode was larger. However, in standard AzerothCore builds, the 4-byte header (2 opcode + 2 length) is the norm.

Another common pitfall is endianness. Ensure that struct.unpack_from('), the length will be interpreted incorrectly, leading to slicing errors that mimic decryption failures.

Session Key Management

Ensure that the key used for decryption is derived correctly from the Diffie-Hellman exchange performed during authentication. If you are storing the key in a global variable, make sure it is updated if the server sends a new key via SMSG_AUTH_SESSION_CHANGED. Stale keys will cause immediate decryption failures on subsequent packets.

For more details on handling cryptographic keys in similar environments, you might find How to Decrypt Private Key in Java: A Comprehensive Guide useful, although the principles of key management remain consistent across languages.

Conclusion

The failure on SMSG_CHAR_ENUM is almost certainly a parsing artifact rather than a cryptographic flaw. By strictly separating the header parsing from the ciphertext extraction and ensuring the RC4 cipher receives exactly the bytes defined by the length field, you will restore synchronization. Always validate the first few bytes of the decrypted payload to confirm structural integrity before attempting to parse complex structures like character enums.

Frequently Asked Questions

Why does SMSG_AUTH_RESPONSE work but SMSG_CHAR_ENUM fail?
Smaller packets like authentication responses are less sensitive to minor byte offsets because the payload is short and may contain repetitive or null-filled data that appears "valid" even when corrupted. Larger payloads like character enums expose the bit-flipping errors caused by RC4 desynchronization immediately, resulting in unreadable garbage data.

Do I need to reset the RC4 cipher for every packet?
Yes, unless the protocol specifies a continuous stream cipher mode. In World of Warcraft, each packet is typically encrypted independently with the same session key. You must create a new ARC4.new(key) instance for each packet decryption to ensure the keystream starts from the beginning (index 0) for every new message.

What if my decrypted data looks like random noise?
Random noise indicates a complete mismatch between the encryption key and the decryption key, or a severe byte offset issue. Verify that the key you are passing to ARC4.new() is the exact byte sequence generated during the DH key exchange, not a hex string or a truncated version of it.

Can I use AES instead of ARC4 for AzerothCore?
No, standard AzerothCore implementations for classic expansions (Vanilla, TBC, WotLK) use ARC4 (RC4) for packet encryption. Using AES would require modifying the server source code to support a different encryption suite, which breaks compatibility with official WoW clients and most existing addons.

How do I handle fragmented TCP packets?
TCP guarantees in-order delivery but does not guarantee message boundaries. You must implement a buffer that accumulates incoming bytes until a complete packet header (opcode + length) is present. Only then should you check if the full payload length is available before attempting to decrypt and parse the packet.