Security researcher Gadi Evron from Knostic reported a tenant isolation bug in the GLM-5.3 Workers AI model on September 8, 2026 (Cloudflare Blog). Cloudflare reproduced the issue, took the model offline within 40 minutes, and re-deployed a patched configuration later that day. No other model hosted on the Workers AI platform was affected by the incident.
The exposure affected 155 responses generated between August 27 and September 9, 2026, all involving prompts exceeding 681,280 tokens. An investigation showed the traffic volume and timing aligned with ordinary long-context user prompts rather than malicious exploitation attempts. Cloudflare confirmed no evidence of intentional misuse was found during its log review.
The incident resulted from a mismatched context window setup alongside an engine handling bug in SGLang's disaggregated mode. Cloudflare deployed the model with a 1,048,576-token context window while the prefill cache pool was limited to 681,280 tokens. Oversized prompts were internally replaced with a single-token placeholder, but prefill generated and sent cache data anyway.
Because the decode node received a transfer completion notification, it assumed the prefill GPU had overwritten the key-value cache with the new context. Instead, the majority of the existing memory buffer remained populated with data from previous client requests. The decode node then generated responses using the retained cross-request memory.
To address the bug, developer Kevin Flansburg submitted pull request #38704 to the sgl-project/sglang repository on September 9, 2026 (GitHub). The fix ensures that admission aborts are preserved before key-value memory handoffs occur. It also implements length validity checks across attention groups before committing output, aborting mismatches with HTTP 500 errors.
Disaggregated serving frameworks like Mooncake separate prefill and decode tasks while managing key-value cache pools across clusters (Github). Mooncake features early rejection logic to reduce wasted computation on requests that fail service level objectives. The SGLang patch specifically added guarded pre-send paths to protect known Mooncake destinations from improper memory states.
Cloudflare's Workers AI catalog maintains multiple active endpoints, including embedding models like BAAI's bge-base-en-v1.5 and earlier generation LLMs like GLM-5.2 (Cloudflare Docs). The platform maintains isolated serving pipelines across its diverse model collection. Systems engineering teams are continuing upstream collaboration to harden disaggregated inference configurations against similar cross-tenant memory bugs.
This incident highlights the subtle security risks inherent in modern disaggregated LLM serving architectures. As inference platforms separate prefill and decode clusters to maximize throughput, state management across memory handoffs becomes a critical security boundary. The interaction between silent input truncation and asynchronous node communication suggests that custom pipeline optimizations require strict, fail-closed validation protocols. Infrastructure providers will likely need to implement standardized sanity checks at the hardware transfer level to prevent stale memory exposure across multi-tenant clusters.
Spotted an error? See our corrections policy or email editorial@lordofclaude.com.
Discussion & Comments (0)
Join the conversation