nodemailer: Quadratic backtracking in SMTP AUTH mechanism detection regexes allows pre-auth remote denial of service

Summary

Nodemailer's SMTP client applies four regular expressions to a peer's EHLO response to detect the supported SASL mechanisms. Each pattern uses an alternation with two overlapping whitespace-matching sub-patterns, which produces quadratic (O(n^2)) backtracking whenever the mechanism name is absent from the line. A malicious or MITM SMTP server that answers EHLO with 250-AUTH followed by a long whitespace-only run and a non-matching token blocks the Node.js event loop for tens of seconds to hours during a pre-authentication handshake. No authentication to the target SMTP server is required, and any application that connects to an attacker-influenced host is affected.

Root Cause

Each of the four AUTH mechanism regexes in SMTPConnection._actionEHLO contains the sub-pattern (?:(\s+|=)[^\n]*\s+|\s+|=) before the mechanism literal. [^\n]* and the trailing \s+ both match whitespace and can consume the same characters, so when the mechanism literal is absent, the regex engine must try every partition of the whitespace run between [^\n]* and \s+ before failing. The four regexes run unconditionally on every EHLO response, with no length cap, no shape validation, and no early rejection before the engine enters the ambiguous alternation.

Component: nodemailer (npm), SMTP client
Function:  SMTPConnection._actionEHLO
Location:  src/smtp-connection/index.ts, lines 2162, 2167, 2172, 2177
Patterns:  /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)PLAIN/i
           /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)LOGIN/i
           /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)CRAM-MD5/i
           /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)XOAUTH2/i

Impact

Algorithmic-complexity denial of service against any Node.js process that opens an SMTP connection to an attacker-controlled or attacker-influenced host. Node.js is single threaded, so the process stalls entirely while the regex engine backtracks: no other requests are served, no timers fire, and the built-in socket timeout (10 minutes) cannot interrupt synchronous JavaScript. A 4,000-character whitespace run inside a single continuation line already blocks each regex for tens of seconds; the client bounds a response at 1 MB, which leaves headroom for blocking runs measured in hours across the four patterns. Two realistic attacker positions reach the sink pre-authentication: a hosted mailer that lets tenants configure their own SMTP server, and any network-adjacent attacker able to man-in-the-middle a cleartext SMTP connection (including before STARTTLS negotiation).

Proof of Concept

The PoC exercises the exact four regexes copied from src/smtp-connection/index.ts against a synthesized 250-AUTH<whitespace>NOMATCH line, so no live SMTP server is needed to reproduce the timing.

Runtime:      Node.js >= 20
Dependencies: none
Invocation:   node poc.js

'use strict';

const AUTH_REGEXES = [
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)PLAIN/i,
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)LOGIN/i,
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)CRAM-MD5/i,
    /[ -]AUTH(?:(\s+|=)[^\n]*\s+|\s+|=)XOAUTH2/i,
];

function bench(n) {
    const line = '250-AUTH' + ' '.repeat(n) + 'NOMATCH';
    const start = Date.now();
    for (const re of AUTH_REGEXES) re.test(line);
    return Date.now() - start;
}

for (const n of [500, 1000, 2000, 3000]) {
    console.log(`n=${n}  four-regex time=${bench(n)} ms`);
}

const control = '250-AUTH PLAIN LOGIN CRAM-MD5 XOAUTH2';
const t0 = Date.now();
for (const re of AUTH_REGEXES) re.test(control);
console.log(`well-formed control  four-regex time=${Date.now() - t0} ms`);

n=500   four-regex time=914 ms
n=1000  four-regex time=1348 ms
n=2000  four-regex time=10558 ms
n=3000  four-regex time=35251 ms
well-formed control  four-regex time=0 ms

Doubling the whitespace run multiplies the wall time by roughly four, which matches the expected O(n^2) growth. The well-formed control shows that a normal EHLO response is parsed in effectively zero time.

Attack Chain

1. Exposure: The victim application, or any dependent that uses nodemailer to send mail, opens a TCP or TLS connection to an SMTP host the attacker controls. This happens automatically for multi-tenant mailers whose tenants configure their own outbound SMTP server, and for any client whose plaintext SMTP session can be intercepted by a network-adjacent attacker before STARTTLS negotiation.
2. Control: The attacker fully controls the bytes returned in response to EHLO. In particular, the attacker chooses the length and content of every 250- continuation line up to the client's 1 MB response cap.
3. Path: Socket bytes are reassembled by _onData, decoded by _processResponse, and dispatched to _actionEHLO. On the first EHLO after the greeting (malicious server with no STARTTLS), on the second EHLO after a successful STARTTLS upgrade, or on the first EHLO when ignoreTLS is set, control reaches the AUTH detection block near the end of _actionEHLO.
4. Guard: The only bounds on this path are maxResponseSize (1 MB by default) and the 421 or non-2xx status-code check. Neither prevents O(n^2) backtracking: 4,000 whitespace characters already block each regex for tens of seconds, well below the 1 MB cap. No length, shape, or sanitization check runs before the regexes.
5. Primitive: Four RegExp.prototype.test calls execute against the crafted line, each entering catastrophic backtracking because [^\n]* and \s+ share the whitespace alphabet and the mechanism literal never matches.
6. Result: The Node.js event loop is blocked for the duration of the four regex evaluations. All other work in the process, including timers, unrelated requests, keep-alive traffic, and the SMTP socket timeout itself, is suspended until the regexes return.

Bypass Evidence

The only response-side protection in this code path is maxResponseSize, introduced by the fix for GHSA-4g23-2xm8-66gc (commit f2d82fa). That fix targeted quadratic reassembly of long multiline replies inside _onData and left the AUTH regex evaluation in _actionEHLO untouched. maxResponseSize does not prevent this issue: at 4,000 whitespace characters, roughly 0.4 percent of the default 1 MB cap, a single continuation line already blocks each of the four regexes for tens of seconds. No status-code or length gate rejects the payload before the regex engine enters the ambiguous alternation, and Node.js's single-threaded event loop cannot service the socket timeout while a synchronous test() call is running. Two other recent linear-time fixes in this project (the addressparser fallback for GHSA-v53p-9fqp-m79j and the DKIM header unfolding for GHSA-39m8-27wv-hr27) address entirely different modules and inputs and do not touch the AUTH mechanism regexes.

Affected Versions

• Ecosystem: npm
• Package: nodemailer
• Confirmed affected range: >= 3.0.0, <= 10.0.12
• Latest release checked: 10.0.12 (npm registry)
• Fix status: not fixed

The four AUTH mechanism regexes were introduced when the SMTP client was bundled into nodemailer at 3.0.0 (previously a separate smtp-connection package). The vulnerable alternation (?:(\s+|=)[^\n]*\s+|\s+|=) is present, unchanged, in every published release from 3.0.0 (lib/smtp-connection/index.js) through 10.0.12 (dist/esm/smtp-connection/index.js and dist/cjs/smtp-connection/index.js). A later refactor added a leading [ -] character class but did not alter the ambiguous inner alternation. No release, and no development branch inspected at the current HEAD (30e4cfd), contains a fix for this pattern.

Suggested Fix

Replace the four backtracking regexes with a non-ambiguous parse of the AUTH extension. The invariant the fix must enforce is that mechanism detection runs in time linear in the response length regardless of whitespace shape. A minimal approach: match the AUTH extension value with a single non-backtracking pattern (for example, /[ -]AUTH(?:\s+|=)([^\r\n]*)/i) and then check for each mechanism name against the whitespace-split tokens with a Set lookup or includes. The fix must cover both continuation forms (250-AUTH ... and 250 AUTH ...) and both delimiters (space and =), and must handle the "AUTH advertised with no mechanisms" case without walking the whitespace run. As an interim mitigation, callers who never connect to an untrusted SMTP peer can restrict SMTP endpoints to a known allowlist; this does not help clients that must interoperate with arbitrary user-configured servers or with any cleartext SMTP path exposed to a network-adjacent attacker before STARTTLS.

---

Reported by zx (GitHub: @manus-pi).