AI Inference as a Service: What It Is and How to Choose a Provider

Training a model is a project. Running it is a bill that arrives every month.

That difference now shapes most AI budgets. Deloitte’s 2026 Technology, Media and Telecommunications Predictions puts inference at about two-thirds of the world’s AI compute in 2026. In 2023 it was about a third. Inference has overtaken training as the larger workload, and Deloitte expects the market for inference-optimized chips to pass $50 billion this year.

The teams feeling this first are the ones that went from pilot to production. A pilot serves a few internal users on a shared API. Production serves customers, at their volumes, with a latency target and a cost attached to every request.

Inference as a service is how most teams cross that line without building GPU infrastructure themselves.

What is AI inference?

Inference is the stage where a trained model does its job. It takes new input and returns an output: a prediction, a classification, a summary, an answer, a transcript.

Training happens once, or a few times, on a large cluster. Inference happens on every request, for as long as the application is live.

That is why the two run on opposite economics. Training is front-loaded and finite. Inference is recurring, and it grows with adoption. For most production applications, inference is where the majority of a model’s lifetime cost lands.

What inference as a service covers

A provider runs the serving infrastructure. You call an endpoint. You supply the model and the traffic. The provider supplies the GPUs, the serving stack, the networking, the scaling, the monitoring, and the people who fix it when it breaks.

Here is the usual split of responsibility.

Three delivery models dominate. The difference between them matters more than the feature lists do.

Most production teams land on the middle column, or move there once traffic becomes predictable.

Why the market moved to inference

Three numbers explain the shift.

Token volume is compounding. OpenRouter routes requests across model providers, so its volume is a useful public proxy for demand. It reported about 25 trillion tokens a week in May 2026, roughly five times its level six months earlier. Agents are a large part of that. One user action can trigger a dozen model calls when an agent plans, calls tools, and checks its own work.

The service market is growing faster than the hardware around it. Precedence Research sizes AI inference as a service at $23.4 billion in 2026, rising to $197.5 billion by 2035. That is a compound annual growth rate of about 26.8 percent. Grand View Research puts the wider AI inference market at roughly $113.5 billion in 2026, reaching about $255 billion by 2030.

Unit prices are falling while total spend climbs. Gartner expects inference on a one-trillion-parameter model to cost providers over 90 percent less by 2030 than it did in 2025. Budgets have still gone up. Gartner also forecasts worldwide AI spending at $2.59 trillion in 2026, a 47 percent increase. Lower prices per token pull more use cases into production, and the total grows anyway.

What actually drives the bill

Published on-demand list prices give the order of magnitude for a GPU-hour. Nebius, one of the providers that publishes a full public rate card, lists H100 at $3.85, H200 at $4.50, B200 at $7.15, and B300 at $7.85 per GPU-hour as of July 2026 (Nebius public rate card, nebius.com/prices, verified 29 July 2026).

The rate card is the easy part. Three things move the bill more than the hourly price does.

• Utilization. A reserved GPU sitting idle costs the same as one under full load. Most inference overspend is capacity bought for a peak that arrives twice a week. Autoscaling behavior and idle-resource controls matter more than the headline rate.
• Batching. Grouping requests together raises total tokens per second and slows each individual answer. Think of a bus against a taxi. The bus moves more people per hour. Any one passenger arrives later. Every serving configuration picks a point on that curve, and the right point depends on whether a person is waiting for the answer.
• Everything billed alongside compute. Storage, data transfer, load balancing, and support are separate lines on most invoices. Ask for a worked example at your expected volume, not a per-hour figure.

Gartner reports that through 2028, at least half of generative AI projects will overrun their budgets. It attributes this to architecture choices and thin operational experience rather than to the price of compute.

Seven things to check before choosing a provider

Inference platforms look alike on a feature page. They behave very differently under production traffic.

• Ask which latency number is being quoted. Tokens per second on a spec sheet is usually measured on an idle machine with a single request. Ask for time to first token and P99 latency at your expected concurrency. A throughput figure taken with an empty queue will not survive contact with real traffic.
• Ask whether the GPUs are yours. On shared capacity, another tenant’s traffic spike becomes your latency spike. Dedicated, single-tenant GPUs make tail latency predictable. That predictability is what user-facing and regulated workloads are really buying.
• Ask what happens to the model behind the endpoint. Managed frontier APIs update models on their own schedule. A version change can move your output quality without your release process knowing. Open-weight models on infrastructure you control stay fixed until you choose to change them.
• Ask how the invoice is built. Per-token pricing is simple until volume grows, and then it scales with your success. Reserved capacity has a floor, and the unit cost falls as volume climbs. Check the metering granularity. Check whether you can attribute spend to a team or a product.
• Ask where the data sits, and under whose law. India’s Digital Personal Data Protection Rules were notified in November 2025, the Data Protection Board is now constituted, and obligations phase in through 2027. For regulated sectors, sending prompts that contain personal or financial data to an endpoint outside the country is a governance question first and a technical one second.
• Ask what the provider does at 2am. Serving stacks fail in specific ways. A node drops. A queue backs up. A quantized model regresses. Read the uptime commitment, the severity definitions, and the escalation path. Check whether support means engineers or a ticket queue.
• Ask how you would leave. Portability comes from standard model formats, standard serving runtimes, and standard APIs. A platform that only runs its own proprietary format makes for a longer conversation later.

Where inference is being used

What makes inference harder than it looks

• Throughput and latency pull against each other. See the bus and the taxi above. There is no configuration that maximizes both.
• Data gravity sets the architecture. Where your data is allowed to live decides where the model can be served, which decides which providers are eligible. Treat it as a design input at the start of the project.
• The security surface is different. A served model accepts untrusted input by design. Prompt injection, jailbreaks, data leakage through responses, and model theft are all inference-time risks. The controls protecting your training pipeline do nothing about them. The OWASP Top 10 for LLM Applications is the standard reference.
• Capacity is uneven. India’s position has improved. The IndiaAI Mission has taken empaneled common compute past 38,000 GPUs, with 20,000 more announced, against 34,333 reported in mid-2025. Even so, confirm that the GPU class you need is available in the region you need, on your timeline.

Where Neysa Velocis fits

Neysa Velocis is an AI acceleration cloud built and run in India, and a SemiAnalysis ClusterMAX Recommended cloud. Teams run production inference on it when their workloads are latency-sensitive, volume-heavy, or supervised by an Indian regulator.

• Dedicated inference endpoints. Models are served on single-tenant GPUs, resident in India. Your tail latency stays yours. Quantization is your choice: FP8, INT4, and AWQ are supported.
• Compute sized to the workload. GPU virtual machines and containers, dedicated bare metal, and the Velocis Kubernetes Engine, which is self-serve and leaves you owning the control plane. Nodes are NVIDIA HGX SXM with NVLink and NVSwitch inside the node. Across nodes, InfiniBand and RoCE v2 run at 3200 Gb/s over an Arista fabric, with WEKA storage underneath.
• Spend you can attribute. Minute-level metering, billing isolation per project, and drill-down on past invoices in the console. Inference cost can be assigned to the team or product that generated it.
• Compliance posture. ISO 27001:2022, SOC 2, ISO 27017:2015, ISO 27018:2019, CSA STAR Level 2, and CSA Trusted Cloud Provider. Data resident in India, aligned to DPDP, RBI, SEBI, IRDAI, and MeitY frameworks. Aegis LLM Shield, Neysa’s standalone LLM security product, can sit in front of a served model and screen prompts and responses against the OWASP LLM Top 10, MITRE ATLAS, and NIST AI RMF.
• Onboarding is assisted. Neysa engineers size the deployment, stand up the environment, and stay with your team through the first production workload. A defined uptime SLA sits behind it. Pricing is quoted in your commercial discussion and depends on GPU class, term, and volume.

Proof. ITQ Technologies runs two production LLM services on Velocis. They interpret fragmented airline fare and refund rules in real time.

“By partnering with Neysa, we have finally moved past the latency and cost barriers of legacy infrastructure to a system that handles our scale with total reliability.”

Anil Porter, Chief Information Officer, ITQ Technologies 

What comes next

Four shifts are worth planning for.

• Agents change the shape of demand. A multi-step agent turns one user action into many model calls. Capacity plans built on request counts will fall short. Plan on tokens and on concurrency instead.
• Serving moves closer to the user. More inference will run at the edge and in-region. Central clusters keep the large models and the batch work. Deloitte’s view is that most inference stays in data centers for now, on the same class of hardware used for training.
• Cost control becomes a platform function. FinOps practice built for cloud spend is being applied to tokens and GPU-hours. Attribution by team and use case is the first step.
• Security moves inline. Screening prompts and responses at the endpoint, rather than auditing after the fact, is becoming standard for regulated deployments.

The common thread: inference stops being an infrastructure decision and becomes an operating discipline. Measured, attributed, governed, and reviewed like any other production system.

FAQ