vllm p2p never connected zmq_exhaustion

Summary

When the OffloadingConnector uses TieringOffloadingSpec with a P2P secondary tier, an API client can submit many unique non-listening peer addresses through kv_transfer_params.prefill. vLLM keeps every never-established P2P session and its ZeroMQ sockets after the request finishes. On vLLM 0.26.0, 340 such peers filled the default ZeroMQ context socket quota; creating the next peer raised an uncaught ZMQError, terminated EngineCore, and shut down the API server. The path was reproduced with a normal API key and no access to the internal P2P listener.

Details

Root cause

In vLLM 0.26.0, P2PSecondaryTierManager.on_new_request() reads the following request-controlled fields:

{
  "kv_transfer_params": {
    "prefill": {
      "kv_request_id": "id",
      "remote_host": "127.0.0.1",
      "remote_port": 49101
    }
  }
}

on_new_request() calls _get_or_create_session(peer_id). This calls ZmqTransport.connect(), creates a P2PSession, and stores it in _sessions before the asynchronous ZeroMQ connection has been established.

ZmqTransport._open_connection() creates a DEALER, enables a libzmq monitor, creates a PAIR socket for that monitor, calls dealer.connect(), and immediately stores the resulting ZmqConnection in _connections. Returning from dealer.connect() does not prove that the peer is reachable.

The monitor subscribes only to zmq.EVENT_DISCONNECTED. A peer that never accepts the initial connection produced EVENT_CONNECT_DELAYED, EVENT_CONNECT_RETRIED, and EVENT_CLOSED, but no EVENT_DISCONNECTED during a 60-second observation. The production monitor therefore never calls mark_dead(), and ZmqConnection.alive remains true.

Request completion calls session.finish_request(), but does not close or remove the peer session. The manager reaps only a session for which session.connected and not session.alive is true. A never-established peer remains connected=True, alive=True, and ready=False. There is no establishment deadline, idle eviction, or limit on pending or total peers.

Each unique retained peer uses about three sockets in the same ZeroMQ context: the DEALER, the monitor endpoint created by libzmq, and the PAIR monitor client. With the measured default MAX_SOCKETS=1023, 340 peers use approximately 1,020 sockets in addition to the local ROUTER. The next peer fails while creating its monitor PAIR. The exception escapes through on_new_request() and the scheduler into the EngineCore loop:

P2PSecondaryTierManager._get_or_create_session()
  -> ZmqTransport.connect()
  -> ZmqTransport._open_connection()
  -> monitor_sock = self._zmq_ctx.socket(zmq.PAIR)
  -> zmq.error.ZMQError: Too many open files
  -> EngineDeadError

The process open-files limit was 65,536. The EngineCore had about 1,215 file descriptors at the failure point, so the failure was not the operating-system FD limit.

Boundary and controls

Each row below used a fresh vLLM process and pre-generated unique non-listening loopback ports.

| Test | Result | |---|---| | Default MAX_SOCKETS=1023, 340 unique peers | 340 HTTP 200 responses; 340 active peers; service healthy | | Default limit, 344 unique peers, concurrency 4 | 340 HTTP 200 and 4 HTTP 500 responses; EngineCore and API exited | | Default limit, 344 unique peers, concurrency 1 | Same 340-peer ceiling and service exit | | Same endpoint repeated 512 times | 512 HTTP 200 responses; one active peer; service healthy | | 512 requests without P2P parameters | 512 HTTP 200 responses; no active peers; service healthy |

A diagnostic A/B changed only the context MAX_SOCKETS value. Raising it from 1,023 to 2,048 moved the last fully successful point from 340 to 682 peers. The 683rd peer then raised the same ZMQError and terminated the service. The approximately 2.01x movement in the boundary identifies the ZeroMQ context quota as the exhausted resource.

The failure runs used logging-only instrumentation around _open_connection() to record the context limit, peer count, and process FD count. It did not add or close any ZeroMQ socket and re-raised the original exception.

Related public work

PR #47843 is an open, unmerged proposal that identifies client-supplied P2P peer addresses as an SSRF risk and proposes API sanitization plus a configured peer allowlist. Those changes would block this public API trigger, but the PR does not add a connection deadline, session cleanup, or a capacity limit for never-established peers. This report adds the deterministic resource-exhaustion and service-exit impact of the retained state.

PR #49823 fixes cleanup and reconnection after a connection has been marked dead. A peer that never establishes does not receive the subscribed EVENT_DISCONNECTED event, so it never enters that dead-state cleanup path.

Impact

This is uncontrolled resource consumption leading to denial of service. An attacker needs network access to an OpenAI-compatible inference endpoint and a normal API key when --api-key is enabled. The attacker does not need access to the internal P2P control or data ports and does not need any peer to answer.

Only deployments that explicitly enable OffloadingConnector, TieringOffloadingSpec, and a P2P secondary tier are affected. Repeating a cheap one-token request with unique non-listening endpoints steadily consumes scheduler and ZeroMQ resources. In the tested default context, about 341 requests were enough to terminate EngineCore and stop all inference on the instance until restart. Confidentiality and integrity were not affected; the demonstrated impact is availability loss.

This advisory was fixed in MR51504.