zlib-rs stack overflow during decompression with malicious inputGHSA-j3px-q95c-9683

A denial of service vulnerability was found in zlib-rs, triggered by specially constructed input. This input causes a stack overflow, resulting in the process using zlib-rs to crash.

Impact

Due to the way LLVM handles the zlib-rs codebase, tail calls were not guaranteed. This caused certain input patterns to result in a large number of stack frames being required, quickly resulting in a stack overflow. These are unlikely to occur in practice, but a dedicated attacker can construct malicious input files.

After stack overflows were found by @inahga with a fuzzer, we dove into the assembly, and found some cases where the stack grew

.LBB109_326:
    mov rdi, rbx
    call zlib_rs::inflate::State::type_do
    jmp .LBB109_311

.LBB109_311:
    lea rsp, [rbp - 40]
    pop rbx
    pop r12
    pop r13
    pop r14
    pop r15
    pop rbp
    .cfi_def_cfa rsp, 8
    ret

LLVM wants to centralize the cleanup before the return (many other blocks jump to LBB109_311), thereby invalidating a tail call to type_do. We were not able to get rid of this call without introducing one elsewhere: we just don't currently have the power to tell LLVM what we want it to do.

So, we switch back to loop+match waiting for changes to rust to make a more efficient implementation possible. Performance-wise, the damage is relatively minimal: we're just slower in cases where we already were slower than C. We are faster in cases where the relevant code is barely touched (in these cases the logic quickly moves into a hot inner loop and just spends most of its time there).

Patches

Version 0.4.0 patches the problem and is no longer vulnerable.

Workarounds

Users of zlib-rs should upgrade to the latest version. Users could alternatively run zlib-rs in a separate process to prevent a stack overflow crashing the entire program. In some situations a signal handler can be used to catch a stack overflow happening.