KV cache leak on Cloudflare Workers AI: how an SGLang bug served other users' data, and how to check your stack
How a KV pool smaller than GLM-5.3's context and an SGLang bug leaked users' data on Workers AI, and how to harden PD serving.
#llm-inference
Every article tagged #llm-inference, across topics.
#llm-inference articles
Mistral Large 4 open weights: the licence, GPU memory, and how to self-host the 1T model
Mistral Large 4 weights are out: what the 1T MoE needs in GPU memory, what the licence allows, and the vLLM and SGLang commands to serve it.
Kimi K3.1 Pro: API, pricing, benchmarks, and what the open weights will need
Kimi K3.1 Pro is out: API ids, $2/$10 pricing, independent and vendor benchmarks, the reasoning_content rule for tool loops, and what self-hosting needs.
DeepInfer explained: how DeepSeek's open-source inference engine serves V4.1 Flash, and what it takes to run it
DeepSeek open-sourced DeepInfer, the engine behind V4.1 Flash: what it includes, what it leaves out, the hardware it needs and how to run it.