CVE-2021-43854 is a high-severity (CVSS 7.5) Uncontrolled Resource Consumption vulnerability in nltk. 3 public exploit references exist, so weaponization risk is real. A fix is available for nltk — see the affected versions and patch details below.
Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize)
EPSS Exploitation Probability
Probability of exploitation in the next 30 days, from FIRST.org EPSS.
How urgent is this, really
CVE-2021-43854 by exploitation likelihood (EPSS) against impact (CVSS). Outside the shaded patch-first corner.
Where this sits among everything scored
Of 385,738 CVEs with a current EPSS score, this one falls in the < 10% band (highlighted). Counts from FIRST.org, log-scaled.
Real-World Exposure
nltkReal-time download stats are indexed for npm and PyPI packages. This vulnerability affects PyPI packages — download data is not available via public APIs for these ecosystems.
Description
Impact
The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack.
In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example:
from nltk.tokenize import word_tokenize
n = 8
for length in [10**i for i in range(2, n)]:
# Prepare a malicious input
text = "a" * length
start_t = time.time()
# Call `word_tokenize` and naively measure the execution time
word_tokenize(text)
print(f"A length of {length:<{n}} takes {time.time() - start_t:.4f}s")
Which gave the following output during testing:
A length of 100 takes 0.0060s
A length of 1000 takes 0.0060s
A length of 10000 takes 0.6320s
A length of 100000 takes 56.3322s
...
I canceled the execution of the program after running it for several hours.
If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.
Patches
The problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result:
A length of 100 takes 0.0070s
A length of 1000 takes 0.0010s
A length of 10000 takes 0.0060s
A length of 100000 takes 0.0400s
A length of 1000000 takes 0.3520s
A length of 10000000 takes 3.4641s
This output shows a linear relationship in execution time versus input length, which is desirable for regular expressions. We recommend updating to NLTK 3.6.6+ if possible.
Workarounds
The execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.
References
- The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866
- The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869
- The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341
- Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html
For more information
If you have any questions or comments about this advisory:
- Open an issue in github.com/nltk/nltk
- Email us at [email protected]
Affected Packages
| Ecosystem | Package | Vulnerable range | Fix |
|---|---|---|---|
| 🐍PyPI | nltk | all versions | 3.6.6pip install --upgrade 'nltk==3.6.6' |
Affected Products
nltknltkResearch use only. For defensive security, authorized penetration testing, and academic research only. Never execute exploit code against systems without explicit written authorization.
Detection & mitigation playbook
Open-source dependencyDetect
Scan your dependency tree (package-lock.json, pnpm-lock.yaml, requirements.txt, go.sum, etc.) for nltk, including transitive dependencies — a direct dependency you never call can still pull in a vulnerable version.
Fix
Update nltk to 3.6.6 or later, then make sure no transitive (indirect) dependency still pins the vulnerable range — O3 confirms CVE-2021-43854 is resolved across your whole dependency graph.
Workarounds
Cap what an attacker can consume: apply request size, rate and timeout limits in front of the affected component, and run it with memory and CPU limits so exhaustion degrades one worker rather than the whole service.
Fixing This On Your OS
If you run this on a Linux distribution, patch through your package manager against the distro's own security advisory below — it tracks the exact backported fix for your release, which can ship on a different timeline (and sometimes a different severity) than the upstream project.
Frequently Asked Questions
Is CVE-2021-43854 in your dependencies?
Find it across PyPI, including transitive dependencies.