OpenAI and Broadcom Unveil a Custom Inference Chip Named Jalapeño

A purpose-built accelerator for serving large language models shows another large buyer designing custom hardware to reduce its reliance on Nvidia for inference.

OpenAI and Broadcom have introduced a custom accelerator, called Jalapeño, built specifically for large language model inference rather than training, according to OpenAI. The case for inference-tuned chips is the standard one: better performance per dollar and per watt on the serving workloads that now dominate a frontier lab's compute costs, since a model is trained once but served continuously.

OpenAI joins a pattern already set by Google's tensor processing units and Amazon's Trainium and Inferentia lines. The economics are straightforward. Inference is a large, predictable, and growing cost, and a buyer at OpenAI's scale can spread the considerable expense of a custom design across enough chips to undercut merchant graphics processing units (GPUs) on its own traffic. Broadcom, which designs the application-specific integrated circuits behind several hyperscaler programs, is the supplier that benefits whether or not any single lab's chip succeeds.

The same day, Nvidia and Amazon Web Services published joint work on production-scale inference covering low-latency serving and GPU price-performance, a reminder that the incumbent is defending the serving layer that custom silicon targets most directly. OpenAI did not disclose the relevant numbers, throughput, latency, and cost per million tokens against a current Nvidia baseline, so the size of any advantage is asserted rather than demonstrated.

What this means

Every large model buyer that designs its own inference chip reduces Nvidia's share of the fastest-growing part of AI spending and shifts value toward custom-silicon designers like Broadcom. The threat is concentrated in inference, where workloads are stable enough to justify a fixed-function part, and far weaker in training, where flexibility still favors general GPUs.

What to watch

Independent benchmarks of Jalapeño's cost and latency against a current Nvidia part, since vendor-stated efficiency gains routinely shrink under third-party testing.
Whether OpenAI shifts a meaningful share of its own serving traffic onto the chip, the only real proof that a custom design is more economical than buying GPUs.

Townsquare

Start a discussion in Townsquare.

Comments

No comments yet.