crawl4ai: ReDoS via untrusted regex patterns freezes the Docker server

Summary

The Docker API server lets an untrusted request body supply regular expressions in three places: RegexExtractionStrategy.custom, the regex field type in a JsonCssExtractionStrategy / JsonXPathExtractionStrategy schema, and RegexChunking.patterns. An authenticated API client can send a pattern with catastrophic backtracking and a page that triggers it. Python's re holds the interpreter lock while it matches, so one request stops the whole server, for all users, for as long as the match runs.

Details

RegexExtractionStrategy and RegexChunking are in UNTRUSTED_ALLOWED_TYPES in crawl4ai/async_configs.py but have no entry in UNTRUSTED_FIELD_ALLOWLIST, so _filter_untrusted_fields() keeps their custom and patterns fields. RegexExtractionStrategy.__init__() compiles each custom pattern and extract() runs it with finditer() over the crawled content. JsonCssExtractionStrategy runs re.search(pattern, value) for every schema field of type regex. RegexChunking.chunk() runs re.split(pattern, ...). None of these bound the pattern or the run time.

Extraction runs in a worker thread, but Python's re engine does not release the GIL during a match, so the event loop cannot serve any other request. The crawl deadline (asyncio.wait_for) cannot interrupt the match either. With the pattern (a+)+$ and a page containing "a" * n + "X", the run time grows about 4x for every 2 more characters: about 1.5 s at n=24, and hours at n=40.

Proof of concept

Server: crawl4ai 0.9.4 Docker image with default settings and a valid API token in $TOKEN.

Serve a page at https://attacker.example/ whose text holds aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaX.

curl -s http://localhost:11235/crawl -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{
  "urls": ["https://attacker.example/"],
  "crawler_config": {"type": "CrawlerRunConfig", "params": {
    "extraction_strategy": {"type": "RegexExtractionStrategy", "params": {"custom": {"evil": "(a+)+$"}}}
  }}
}'

Result: the request is accepted. The match backtracks exponentially, and the whole server stops answering, including /health, because re holds the GIL. The crawl deadline does not stop it.

The same works with RegexChunking ("chunking_strategy": {"type": "RegexChunking", "params": {"patterns": ["(a+)+$"]}}) and with a "type": "regex", "pattern": "(a+)+$" field in a JsonCssExtractionStrategy schema.

Library-level timing, no network: with 24 a characters each case takes 1 to 3 seconds, and every 2 more characters is about 4 times slower.

Impact

Any authenticated API client can make the Docker server stop answering every request, including health checks, with one /crawl request. No optional extras are needed. Confidentiality and integrity are not affected.

The library used directly is not affected unless an application passes untrusted regular expressions into these strategies.

Patch

Untrusted request bodies may no longer supply regular expressions: RegexExtractionStrategy.custom, RegexChunking.patterns, and schema fields of type regex in JsonCssExtractionStrategy, JsonXPathExtractionStrategy and JsonLxmlExtractionStrategy are refused. The built-in RegexExtractionStrategy patterns stay available. Fixed in 0.9.5.

Workaround

At your reverse proxy, reject crawler_config bodies that contain RegexExtractionStrategy, RegexChunking, or a schema field with "type": "regex". Give API tokens only to trusted clients.

Credit

Reported by Zhixi "Jace" Sun (manus-use), independent security researcher.