Protego has exponential backtracking ReDoS in robots.txt URL wildcard matchingGHSA-wjmf-p669-5m5p
Fix: scrapy/protego@7859401GHSA-wjmf-p669-5m5p is a Uncontrolled Resource Consumption vulnerability in protego. A fix is available for protego — see the affected versions and patch details below.
Exploitation Status
No confirmed exploitation observed yet
- CISA assesses this as automatable — exploitation doesn’t require manual, per-target effort, which raises the odds of mass scanning and opportunistic attacks.
- CISA’s own triage has not observed active exploitation or public proof-of-concept code for this CVE as of its last assessment.
Exploitation and automatability from CISA’s SSVC triage for GHSA-wjmf-p669-5m5p.
EPSS Exploitation Probability
Probability of exploitation in the next 30 days, from FIRST.org EPSS.
Real-World Exposure
protegoReal-time download stats are indexed for npm and PyPI packages. This vulnerability affects PyPI packages — download data is not available via public APIs for these ecosystems.
Description
Problem description
Protego constructs regular expressions to match URLs against robots.txt Allow: and Disallow: directives, see protego._urlpattern._URLPattern._prepare_pattern_for_regex(). Every * in the directive value is translated into a lazy .*? regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
Impact
Parsing a specially crafted robots.txt with protego.Protego.parse() and then trying to match an URL with protego.Protego.can_fetch() results in the latter call not returning for a period dependent on the length of the URL.
Proof of concept
from protego import Protego
robotstxt = f"""
User-agent: *
Disallow: /{"*1" * 12}*Z
"""
rp = Protego.parse(robotstxt)
url = "/" + "1" * 60
rp.can_fetch(url, "mybot") # freezes
Affected Packages
| Ecosystem | Package | Vulnerable range | Fix |
|---|---|---|---|
| 🐍PyPI | protego | all versions | 0.6.2pip install --upgrade 'protego==0.6.2' |
Detection & mitigation playbook
Open-source dependencyDetect
Scan your dependency tree (package-lock.json, pnpm-lock.yaml, requirements.txt, go.sum, etc.) for protego, including transitive dependencies — a direct dependency you never call can still pull in a vulnerable version.
Fix
Update protego to 0.6.2 or later, then make sure no transitive (indirect) dependency still pins the vulnerable range — O3 confirms GHSA-wjmf-p669-5m5p is resolved across your whole dependency graph.
Workarounds
Cap what an attacker can consume: apply request size, rate and timeout limits in front of the affected component, and run it with memory and CPU limits so exhaustion degrades one worker rather than the whole service.
Fixing This On Your OS
If you run this on a Linux distribution, patch through your package manager against the distro's own security advisory below — it tracks the exact backported fix for your release, which can ship on a different timeline (and sometimes a different severity) than the upstream project.
This Moderate denial of service flaw in Protego, a robots.txt parser, does not affect Red Hat products as the vulnerable component is only shipped in Fedora. This vulnerability has been fixed in upstream version 0.6.2.
Mitigation for this issue is either not available or the currently available options do not meet the Red Hat Product Security criteria comprising ease of use and deployment, applicability to widespread installation base or stability.Source: Red Hat security advisory for GHSA-wjmf-p669-5m5p (CC BY 4.0)
Frequently Asked Questions
Is GHSA-wjmf-p669-5m5p in your dependencies?
Find it across PyPI, including transitive dependencies.