KV cache leak on Cloudflare Workers AI: how an SGLang bug served other users' data, and how to check your stack

For thirteen days, Cloudflare's Workers AI endpoint for GLM-5.3 could answer a very long prompt with text from someone else's request. Cloudflare's incident write-up, published on 26 October 2026, counts 155 affected responses between 27 August and 9 September, all to prompts longer than 681,280 tokens. The model was not at fault: a context window bigger than the prefill KV cache pool met a bug in SGLang's disaggregated prefill/decode path, and decode read cache pages that still held earlier users' data.

This article explains the mechanism in plain terms, what SGLang changed upstream in its companion security post, and what to check if you run SGLang (or any paged-KV inference engine) in prefill/decode mode. It is for engineers who operate LLM inference, and for teams sending sensitive prompts to shared inference APIs who want to know what this class of bug looks like.

What happened in the Cloudflare Workers AI KV cache incident?

Some Workers AI customers sending very long prompts to GLM-5.3 received coherent answers built partly from other customers' recent prompts and generations. Cloudflare's own summary is that "a small number of customers using the GLM 5.3 Workers AI model received responses that contained data from other customers' requests" (Cloudflare).

Two things had to be true at once. GLM-5.3 was deployed with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs held only 681,280 tokens. Cloudflare knew about the mismatch before launch; its long-context tests got HTTP 200s with no error frames, so it was not treated as a problem. The post calls this a testing gap: the tests checked status codes and performance, not what the model actually said. The second ingredient was an SGLang bug that turned "this prompt is too big" into "run it anyway on someone else's cache pages".

After the fix, Cloudflare aligned GLM-5.3's configuration with GLM-5.2, which freed GPU memory: the prefill pool now holds about 2.04 million tokens, roughly twice the context length, so every valid prompt fits.

How the SGLang prefill/decode bug leaked other users' data

In short: a request that should have been rejected was admitted, prefill wrote one KV page for it, decode reserved pages for the whole prompt, and attention read the unwritten pages, which still held other people's data. Here is the path, from Cloudflare's root-cause section and SGLang's issue #30233.

1. 1

   Prefill rejects the prompt, but only on paper

   When a prompt does not fit the KV pool, SGLang marks it rejected with Req.set_finish_with_abort and swaps the prompt for a one-token placeholder. On a single-node deployment that path returns an error to the client.

2. 2

   The admission queue checks the wrong flag

   In disaggregated mode, the admission queues let in any request where req.finished() was False. A request with a pending rejection is not yet finished, so it was admitted. Prefill ran a forward pass on the placeholder, sent one KV page to decode and reported the transfer complete. Issue #30233 adds that the in-flight queue then overwrote the abort status with FINISH_LENGTH(length=0).

3. 3

   Decode reserves the full prompt and reads stale pages

   The router had told decode the real prompt length, eight pages in Cloudflare's diagram, and decode reserved eight. Only P0 had been written by this request. In Cloudflare's words: "Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it." P1 to P7 still held earlier requests' prompts and generations, attention read them as context, and the client got a fluent answer with an HTTP 200.

The bug was not unknown. Issue #30233, filed on 6 July 2026 against SGLang 0.5.11 with GLM-5.2-FP8 on Mooncake, describes the same thing: a prompt just over max_req_input_len makes prefill report prompt_tokens=1 and decode return unrelated content. Cloudflare says it had been flagging KV-related issues to SGLang since v0.5.10.

The second SGLang bug: an RDMA abort race

SGLang's post describes a separate, broader race that affects any PD deployment with a KV transfer backend, not only oversized prompts. KV moves from prefill to decode as asynchronous one-sided RDMA WRITEs. If a request aborts mid-transfer, decode can free its slots while writes are still in flight; the allocator hands those slots to a new request, and the late writes land in that request's cache (SGLang).

Posted RDMA work requests cannot be recalled and the receiving CPU gets no per-write signal, so the fix is to hold the slots until prefill confirms the transfer has drained, or a timeout passes. SGLang states the required hold as:

Long prompts make it worse on every term: a longer prefill widens the in-flight window, lengthens the tail after an abort, and raises abort rates. The leak runs one way: fragments of the aborted request's context can appear in another client's output.

What Cloudflare and SGLang changed

Cloudflare took the model private at 14:00 UTC on 9 September, about 40 minutes after declaring the incident, rolled every prefill node to a corrected configuration by 14:52, rebuilt the deployment with an engine patch by 19:04 and restored public access at 19:40. Its patch retires a request with a pending rejection at the disaggregation admission queue, returning the original error before any bootstrap, allocation or KV transfer; it went upstream as #38704.

• Admission: #36988 retires aborted requests after their final prefill result; #38935, merged 16 September, rejects them at intake before any handoff state exists.
• Deferred KV release: decode holds slots until a drain acknowledgement or timeout, gated per backend (#37454, #41023) and on by default in v0.5.21.
• Bounded Mooncake batches: #32758 splits transfers into capped batches.
• Integrity check: #39500 adds an opt-in Adler-32 checksum on PD transfers.
• Metrics: #41128 adds sglang:decode_deferred_kv_release_total{outcome} in v0.5.22.
• Load balancer: Cloudflare changed its proxy so a client disconnect no longer aborts a request mid-transfer, and added deployment checks comparing KV pool size with context length.

Longer term, Cloudflare is working with SGLang on per-page integrity tags checked at read time. A prototype cost under 1% in throughput and tail latency but was disabled in August for reliability reasons, according to the post.

How to harden an SGLang prefill/decode deployment

Upgrade first; everything else is defence in depth. On v0.5.21 or later the deferred release is on by default for Mooncake and NIXL. On v0.5.19 or v0.5.20, SGLang's stopgap is an environment-variable hold on both prefill and decode, with a 60–120 s timeout. The flags and variables below are the ones named in SGLang's post and the PD disaggregation docs.

# Hold decode KV slots until prefill confirms the transfer drained (both roles)export SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1export SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT=90   # seconds, T_hold# Cap Mooncake transfer batches (0 = whole chunk in one call); costs throughput and TTFTexport SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES=4096 python -m sglang.launch_server \  --model-path "$MODEL" \  --disaggregation-mode prefill \  --disaggregation-transfer-backend mooncake \  --disaggregation-ib-device mlx5_roce0 \  --disaggregation-enable-kv-checksum \  --port 30000

Then size the pool. The single question that would have prevented the Cloudflare incident is: can the prefill KV pool hold the configured context length? SGLang reports its pool capacity as max_total_num_tokens (the field quoted in issue #30233); compare it with --context-length after every change to model, quantisation, mem_fraction_static or parallelism. Finally, put a PD-aware router in front, such as sgl-router in PD mode, which cancels both halves of a prefill/decode pair on disconnect (#41612), and watch the release metric: a healthy deployment releases mostly through drained, and a high timeout share means the acknowledgement path needs attention.

Test any LLM endpoint for cross-request leaks with canaries

Status codes will not tell you whether outputs are isolated; Cloudflare's launch tests passed on HTTP 200s. A canary test checks content. Send many concurrent requests, each with its own random marker, and fail if any response contains a marker from a different request. It runs against any OpenAI-compatible endpoint, including SGLang's own server and the router, using only the Python standard library.

"""Cross-request leak check for an OpenAI-compatible endpoint. Each request carries its own random canary. If any response contains acanary that belongs to a *different* request, the server mixed state."""import json, os, secrets, sys, urllib.requestfrom concurrent.futures import ThreadPoolExecutor BASE = os.environ.get("BASE_URL", "http://localhost:8000/v1")KEY = os.environ.get("API_KEY", "")MODEL = os.environ.get("MODEL", "default")ROUNDS = int(os.environ.get("ROUNDS", "20"))PAD_WORDS = int(os.environ.get("PAD_WORDS", "2000"))  # raise towards your context limit def ask(canary: str) -> str:    padding = " ".join(["filler"] * PAD_WORDS)    prompt = f"{padding}\nThe secret code is {canary}. Reply with one short sentence about the weather."    body = json.dumps({"model": MODEL, "max_tokens": 64,                       "messages": [{"role": "user", "content": prompt}]}).encode()    req = urllib.request.Request(f"{BASE}/chat/completions", data=body, headers={        "Content-Type": "application/json", "Authorization": f"Bearer {KEY}"})    try:        with urllib.request.urlopen(req, timeout=300) as r:            return json.load(r)["choices"][0]["message"]["content"] or ""    except urllib.error.HTTPError as e:        return f"HTTP {e.code}"  # a refusal is the correct answer to an oversized prompt canaries = [f"CANARY-{secrets.token_hex(8)}" for _ in range(ROUNDS)]with ThreadPoolExecutor(max_workers=8) as pool:    answers = list(pool.map(ask, canaries)) leaks = [(mine, other) for mine, text in zip(canaries, answers)         for other in canaries if other != mine and other in text]for mine, other in leaks:    print(f"LEAK: response to {mine} contains {other}")print(f"{len(canaries)} requests, {len(leaks)} cross-request leaks")sys.exit(1 if leaks else 0)

BASE_URL=http://localhost:8000/v1 MODEL=my-model ROUNDS=50 PAD_WORDS=200000 python3 canary_check.py

Push PAD_WORDS past your pool size as well as below it: an oversized prompt should come back as an HTTP 4xx, never as a fluent answer. A clean run is evidence, not proof. The abort race is probabilistic, so run it under realistic load and with clients that disconnect mid-request.

What this means if you only call an inference API

If you are a Workers AI customer, Cloudflare's post gives no per-customer instructions and does not describe notifying affected customers; ask your account contact if you sent prompts over 681,280 tokens to @cf/zai-org/glm-5.3 in the window. More generally, shared inference is multi-tenant GPU memory, and this was an allocation bug, not a model behaviour. Treat prompts as data that can leak: keep secrets and raw .env contents out of them, prefer providers that publish incident write-ups like this one, and use dedicated deployments for workloads that cannot tolerate cross-tenant exposure.

Checklist for SGLang operators

• To do: Upgrade SGLang to v0.5.21+ (Mooncake, NIXL) or v0.5.22+ (MORI); v0.5.23 for failed-send protection on NIXL and MORI.
• To do: Compare the prefill and decode KV pool size with the configured context length on every node, after every config change.
• To do: On v0.5.19/v0.5.20, enable the deferred decode KV release on both roles with a 60–120 s timeout.
• To do: Run a PD-aware router that cancels both halves of a pair, and stop your proxy from aborting requests on client disconnect.
• To do: Consider --disaggregation-enable-kv-checksum, and watch sglang:decode_deferred_kv_release_total{outcome}.
• To do: Add a canary test that checks response content, not status codes, to your launch and regression suites.

Sources and further reading

• Cloudflare: Cross-request KV cache exposure in the GLM-5.3 Workers AI model (Kevin Flansburg and Michelle Chen, 26 October 2026) and the status page incident
• SGLang / LMSYS: Security update: improving request isolation and the reproduction gist
• SGLang issue #30233 and fixes #36988, #38704, #38935, #41128
• SGLang PD disaggregation documentation and Mooncake
• Hacker News discussion