B300 in Production: What 18 Benchmark Runs Actually Showed
In two weeks this summer, OpenAI unveiled Jalapeño, its first inference chip, and Reuters reported that DeepSeek is designing one too. Reuters puts Anthropic among the handful of labs doing the same. The big cloud providers already have theirs: Google Ironwood, AWS Trainium, Microsoft Maia. And a dozen startups are selling silicon built for one job, running models: Groq, Cerebras, SambaNova, d-Matrix, Etched.
Why the rush? Inference is now the bigger bill. Deloitte puts inference at about half of AI compute in 2025 and two-thirds in 2026. Menlo Ventures reported 74% of builders say most of their workloads are inference, up from 48% a year earlier. When the recurring cost moves, the hardware follows it.
So the pitch lands in your inbox every week: a specialist chip that serves tokens faster and cheaper than the graphics processing unit (GPU) you are on. The benchmark behind it is usually real. It just may not hold for your workload. That is the catch with specialized silicon.
The right choice is the one that fits what you are actually running, not the one that wins the benchmark. So start with your use case and work back to the silicon.
Serving a model is a different job from training one
Training and inference stress a chip in opposite ways. That is the whole reason inference chips exist.
Answering a prompt happens in two phases. Prefill reads your prompt. It is compute-bound: lots of dense math done in parallel. Then decode writes the answer one token at a time, and it is memory-bound. Every new token reads the whole model and the running context out of memory before it can appear. For most chat and agent workloads, decode is where the time and the cost sit.
That split rewires the spec sheet. For serving, memory capacity and memory bandwidth matter more than the peak training floating-point operations per second (FLOPS) at the top of the page. It is why every 2026 inference part leads with low-precision number formats. FP8 roughly halves what a model weighs. FP4 and INT4 cut it again, so more of the model moves per second. The split is already showing up in production. In July, inference provider Parasail put d-Matrix Corsair accelerators alongside NVIDIA GPUs, running prefill on the GPU and decode on the cheaper specialist silicon.
Memory capacity matters for a blunt reason too. A frontier model may not fit on one chip. Llama 3.1 405B needs about 810GB at 16-bit precision, and the largest single accelerators today hold 288GB. When a model does not fit, you chain chips together and pay for the network between them. That is the memory wall every inference-chip company is really fighting.
The 2026 landscape: a chip for every job
Step back from the logos and the field sorts into four buckets.
Memory-rich GPUs carry the whole model on one card and run anything. The AMD Instinct MI350X and NVIDIA B300 both hold 288GB; the H200 holds 141GB. These are the general-purpose default.
Wafer-scale and dataflow chips trade generality for raw speed on a served model. Cerebras runs a model out of 44GB of on-chip memory. Groq’s language processing unit (LPU) is built the same way: all static memory (SRAM) on the chip, none off it, tuned for fast single-stream decode. SambaNova sits between the two with a three-tier memory design.
Cloud-provider silicon is the big clouds’ own answer: Google’s Ironwood tensor processing unit (TPU) with 192GB, AWS Trainium, Microsoft Maia. You rent it inside their cloud rather than racking it in yours.
Purpose-built ASICs, or application-specific integrated circuits, go furthest. d-Matrix keeps compute and memory on the same chip and skips scarce high-bandwidth memory (HBM) entirely. Etched went further and hard-wired the transformer into the silicon: fast on the models it was built for, useless on anything else.
| Category | Examples | Memory | Best-fit workload | Portability / lock-in |
| Memory-rich GPUs | AMD MI350X, NVIDIA B300, H200 | 141-288 GB HBM3E | Any model, general serving | High. PyTorch/ROCm/CUDA, day-zero models |
| Wafer-scale & dataflow | Cerebras WSE-3, Groq LPU, SambaNova | 44 GB SRAM to multi-tier | Fast single-stream decode | Medium. Own software kit (SDK), usually a cloud endpoint |
| Cloud-provider silicon | Google TPU Ironwood, AWS Trainium, MS Maia | up to 192 GB HBM3E | High-volume serving in that cloud | Low. One cloud, its own toolchain |
| Purpose-built ASICs | d-Matrix Corsair, Etched Sohu | In-memory / 144 GB HBM3E | One stable, high-volume model | Low. Etched runs transformers only |
Two things are true here at once. The engineering is good, and the independent numbers are closing on the leading GPUs. On the audited MLPerf v5.1 round, the specialist gap to the best GPUs is measured in tens of percent, not multiples. Even so, a GPU still runs any model the day it ships. That is why NVIDIA still holds around 80% of the accelerator market, and closer to 60 to 75% in inference specifically.
Score a serving chip on five things, not one
| Axis | The question | Why it decides your bill |
| Fit | Does the model plus its context fit in memory? | If it spills, you chain chips and pay for the network between them |
| Feed | Is there enough memory bandwidth to keep it busy? | Bandwidth, not FLOPS, sets your per-token speed on decode |
| Cost | Cost per million tokens at your batch size? | The benchmark’s batch is not the batch you will run |
| Latency | Time to first token and speed one user feels? | The aggregate hides what a person actually experiences |
| Portability | Can you move off it? | The model you serve next quarter may not run on it |
No chip wins all five for every workload. A wafer-scale part may top Latency for one user and lose on Cost at scale. An ASIC may win Cost on a stable model and score near zero on Portability.
That is the point of a scorecard. It stops you buying on the one number the seller chose.
A peak number is measured at a batch size you will never run
The gap between a spec sheet and production comes down to one thing: batching.
A chip hits its peak tokens per second by packing many requests together. That keeps every circuit busy. Your one user gets a slice of it. Here is the same NVIDIA B200, in NVIDIA’s own InferenceMAX numbers, reported two ways.
| Model | Aggregate, per GPU (the headline) | Per user (what a person feels) |
| gpt-oss | 60,000 tok/s | 1,000 tok/s |
| gpt-oss, speculative decode | 30,000 tok/s | 100 tok/s |
| Llama 3.3 70B | over 10,000 tok/s | 50 tok/s |
Same silicon. The headline is measured where the chip is fullest. You will run where your p99 latency, the slowest 1% of requests, stays inside your service level agreement (SLA). That point is far lower. Push a GPU’s batch size up and throughput can rise 14x while latency gets 4x worse. Run at batch size one to keep a single user fast, and you leave most of the GPU idle. You do not get to pick both ends of that slider.
The other thing the spec sheet will not tell you: software moves your cost more than the chip does. NVIDIA cut a model’s token cost about 5x in a month through serving-stack optimization alone, on hardware that did not change. SemiAnalysis’s InferenceMAX shows the same swing from the latency target you hold the chip to. Four things move your cost per token. Only one of them is the chip.
| Move | Cost per token | The trade-off |
| Bigger batches | falls | every user’s answer slows down |
| Tighter latency target | rises | you serve fewer requests at once |
| Better serving software | falls sharply | the work is in the tuning |
| A faster chip | falls a little | the smallest lever of the four |
The chip sets the floor. The software running the queue sets the ceiling. Most teams shop hard for the chip and never touch the software.
The model you serve next year is not the one you serve today
The field is moving fast, and prices with it. The cost to serve a fixed level of quality has fallen roughly 10x a year. To hold a GPT-4 level of performance, Epoch clocks it at up to 40x. A frontier open model lands every few weeks. The one you pick today will look dated within a year.
On a GPU from AMD or NVIDIA, a new model runs from day zero: PyTorch, vLLM and SGLang ship recipes the week it drops. On a chip with its own toolchain, you wait for support. Etched’s design runs transformers and nothing else.
That is the real cost of a specialist chip, and it never shows up on the spec sheet.
| When a dedicated ASIC wins | When a GPU wins |
| One stable model, served at huge volume | A model roadmap that changes every few weeks |
| A fixed latency target you tune once | New architectures you need to adopt day-zero |
| Cost per token and power are the whole game | Portability and a standard toolchain matter |
| You can absorb the software and lock-in cost | You want to run your model anywhere |
If you serve one stable model at massive scale, an ASIC’s cost-per-token and power savings can dominate. That is why analysts expect the largest clouds to move most internal inference to custom silicon by the end of the decade. But it is a bet that your workload holds still, and most workloads move. Teams that keep their options open treat portability as a feature they will use. Tenstorrent has built a whole open-source stack around exactly that pitch.
On a GPU, switching models is a base-image change. On the wrong ASIC, it means buying new hardware.
How to actually choose
Do not start with the chip. Start with the workload.
Write down the model and how big its context gets. Write down your traffic shape and your latency SLA. Now the five axes have values instead of adjectives. Does the model fit? Does your SLA leave room to batch? Is your model roadmap fixed for the next year? The answers tell you whether you are shopping for a memory-rich GPU, a low-latency specialist, or a stable-model ASIC.
Then run your own model on the shortlist. A benchmark deck is measured on someone else’s workload at someone else’s batch size. Yours is the only one that counts.
If you would rather not run the chips yourself
There is a middle path between renting a raw GPU and buying a rack of new silicon: let someone else operate the serving layer. That is what Neysa Velocis offers in India. Managed inference as a service, on dedicated single-tenant endpoints, running on leading AMD and NVIDIA GPUs. Neysa runs the endpoint. You put your model on either vendor and compare cost and speed on your own traffic before you commit.
You do not need to own the chip. You need the right one for the job, run well. Score the five axes, run your model, and leave the headline number for last.
