Nvidia Puts Groq 3 LPX Into Full Production and Claims 30x More Agent Throughput Per Megawatt

Nvidia used the Hot Chips 2026 conference to move its inference roadmap from announcement to shipment. The company said its Groq 3 LPX accelerator has entered full production, a latency-focused extension of the Vera Rubin data center platform, eight months after it signed a $20 billion licensing agreement with the chip designer Groq. CNBC reported that the first racks will be online before the end of this year, deployed alongside Vera central processing units and Rubin graphics processing units (GPUs) at Nebius, a neocloud (a cloud-computing provider that specializes in renting out AI infrastructure), through its Token Factory platform.

The architecture divides the inference task into two parts. Vera Rubin NVL72 handles large-scale context processing. The Groq 3 LPX rack, which StorageReview describes as 256 language processing units (LPUs) per liquid-cooled rack reaching 3,400 tokens per second at a 100,000-token context, handles the latency-critical work of decoding tokens. Nvidia told CNBC that an LPX rack delivers up to 35 times more inference throughput per megawatt. The system also includes Spectrum-X Multiplane, an Ethernet fabric Nvidia says scales to 512,000 GPUs, BlueField-4 data processing units, and NVLink Fusion, a technology that lets third-party custom accelerators attach to Nvidia's sixth-generation NVLink rack systems.

The headline efficiency figure requires scrutiny. Nvidia says Vera Rubin NVL72 delivers up to 30 times higher throughput per megawatt than GB300 NVL72 on agentic workloads, but that figure comes from Nvidia running the SemiAnalysis AgentX workload itself, not from an outside lab. Nvidia attributes the gain to system-level engineering rather than the chip alone: mixture-of-experts serving runtimes (SGLang, TensorRT-LLM, vLLM), DeepGEMM kernels, the MXFP4 and MXFP8 mixed-precision formats, and the Dynamo session-aware serving stack.

The motivation is the shape of demand. Nvidia cites OpenRouter data showing that agentic workloads consume 15 times more tokens than a simple chat request, because each step's accumulated output becomes the input for the next step. That makes long-context prefill and fast decoding the two costs that determine whether an agent product is profitable.