Autobahn Python: permessage-brotli inflates a whole frame before maxMessagePayloadSize is checked (incomplete fix of GHSA-hxp9-w8x3-p566)GHSA-pjh6-c8mg-3x7m

Summary

permessage-brotli inflates each frame fully before checking maxMessagePayloadSize, so one 421-byte frame makes the server allocate about 545 MiB even with a 1 MiB message limit (brotli expands about 1,000,000:1). Any unauthenticated client can exhaust server memory this way on servers that enable permessage-brotli. This bypasses the GHSA-hxp9-w8x3-p566 fix.

Details

In 26.7.1 the receive path passes the remaining maxMessagePayloadSize budget to decompress_message_data(..., max_output_len=...). deflate and bzip2 stop at that budget. PerMessageBrotli.decompress_message_data (compress_brotli.py) instead calls self._decompressor.process(data) with no limit and compares the length afterwards, so the whole frame is inflated first.

The code comment and changelog say this is still "bounded on the wire by maxFramePayloadSize", but brotli reaches about 1,000,000:1 (841 bytes inflate to 1 GiB), so that bound means little. A 421-byte frame against a 1 MiB budget peaks at 545 MiB before the 1009 rejection.

An autobahn asyncio server on 127.0.0.1 accepting permessage-brotli, with maxFramePayloadSize=64 KiB and maxMessagePayloadSize=1 MiB, received one small brotli frame from an autobahn client and peaked at 545 MiB (baseline 28 MiB). A few concurrent connections can exhaust memory.

brotli ≥ 1.2.0 has Decompressor.process(data, output_buffer_limit=N), so the backend can stop at max_output_len like deflate and bzip2 (and the extra's floor can be raised to brotli>=1.2.0).

Tested on autobahn 26.7.1 (latest) with brotli 1.2.0; compress_brotli.py on master is unchanged. This affects servers that enable permessage-brotli (opt-in, needs the compress extra).

Proof of concept

How to run the PoC

python3 -m venv venv && . venv/bin/activate
pip install autobahn==26.7.1 brotli==1.2.0
python3 poc_autobahn.py

# PoC: autobahn 26.7.1 permessage-brotli inflates a whole frame before maxMessagePayloadSize is checked.
import brotli, resource
from autobahn.websocket.compress import PerMessageBrotli

c = brotli.Compressor()  # build a 256 MiB-of-zeros frame without holding the plaintext
frame = b"".join(c.process(b"\0" * (1 << 20)) for _ in range(256)) + c.finish()
pmce = PerMessageBrotli(is_server=True, server_no_context_takeover=False, client_no_context_takeover=False)
pmce.start_decompress_message()
print(len(frame), "bytes on the wire; peak RSS before:", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss >> 10, "MiB")
try:
    pmce.decompress_message_data(frame, max_output_len=1 << 20)  # remaining budget: 1 MiB
except Exception as e:
    print(type(e).__name__, "| peak RSS after:", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss >> 10, "MiB")

Output:

421 bytes on the wire; peak RSS before: 36 MiB
PayloadExceededError | peak RSS after: 545 MiB

Impact

An unauthenticated WebSocket client can make a server that enabled permessage-brotli allocate hundreds of MB or more per tiny frame, regardless of the configured maxMessagePayloadSize. That is the memory-exhaustion DoS GHSA-hxp9-w8x3-p566 was meant to prevent.

Credit goes to Adrian Korwel.