GraphQL-core: Denial of service via comment tokens bypassing the parser's max_tokens limit

Summary

The parser's max_tokens limit — the documented defence against oversized GraphQL documents — did not count comment tokens. A document padded with comments could therefore pass a max_tokens check of any size while still allocating one Token object per comment, all of which stay reachable for the lifetime of the parsed document. A 38 MB comment-padded payload allocated roughly 92 MB of additional resident memory with max_tokens=5 in force.

Details

Lexer.lookahead() reads tokens one at a time and skips over comments, returning only the first non-comment token. Each skipped comment is nevertheless fully tokenized, allocated as a Token object, and linked into the persistent token list via token.next / token.prev — the chain that powers source-location reporting, and which is retained as long as the document is.

Parser.advance_lexer() incremented the token counter only for the tokens the lexer actually returned. Comments were therefore invisible to the accounting, and max_tokens bounded only the *non-comment* tokens.

The practical consequence is a gap between what users reasonably believe max_tokens protects and what it did protect. The limit was not broken for ordinary documents — real tokens were still capped correctly — so the shortfall was not observable without specifically testing comment padding.

As with the companion issue, the payload compresses extremely well (# c\n repeated), so body-size limits and WAF inspection applied to the compressed request do not see its true size, while the framework decompresses it in memory before the parser is reached.

Proof of concept

import gc
import resource
from graphql import parse

# 100,000 comments, a 5-token query, and max_tokens=5 — accepted.
query = "# c\n" * 100_000 + "{ a }"
parse(query, max_tokens=5)          # no error raised

# Memory amplification.
big = "# padding comment used to amplify memory per token\n" * 500_000 + "{ a }"
gc.collect()
before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
parse(big, max_tokens=5)            # ~38 MB source
after = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(f"RSS delta: +{(after - before) / 1024:.0f} MB")

Measured on GraphQL-core 3.2.8 / CPython 3.12 / Linux x86_64: the 391 KB payload parsed in 127 ms, and the 38.1 MB payload produced an RSS delta of +92 MB — about 2.4× the source size, on top of the source string itself.

Impact

Denial of service through memory exhaustion. A modest number of concurrent requests can push a worker past a typical 256 MB or 512 MB container limit and trigger an OOM kill. No authentication is required, and no data is disclosed or modified.

Depth limiters and complexity analyzers do not help: they run on the AST after parsing has already succeeded, and the document contains only five real tokens.

Patches

Fixed in 3.2.12 and 3.3.0rc1.

Parser.advance_lexer now counts every token consumed, including the comment tokens the lexer skips between the previously focused token and the new one, and including the advance onto EOF so that trailing comments are covered. max_tokens therefore now bounds every Token the parser allocates.

Note the behaviour change: **comments now count toward max_tokens.** A comment-heavy document that was previously accepted may now be rejected. If you parse documents containing substantial comments — SDL files with documentation comments, for instance — and you set max_tokens, you may need to raise the limit. The token_count attribute of a parsed document likewise now includes comments.

Workarounds

If you cannot upgrade, apply a request body-size limit to the decompressed body before the document reaches parse().

Such a limit is worth keeping after upgrading as well: max_tokens is one guard among several and bounds the token count, not the overall cost of a request.

References

• GraphQL specification § 2.1.4 — Comments

Credits

Reported by @BZHunt, who also reviewed the fixes before release. In memory of ZaX, who was the driving force behind this discovery.