We spent 20 working days getting one model into production. So we built Managed Endpoints.

Over the last several months, we have worked with a number of enterprises that wanted to try open models. Not a chatbot demo. Real work: coding agents, internal tools, pipelines that run all day.

Every one of those engagements followed the same script. The team picks a model. They spin up a VM, set up vLLM or SGLang, and then start answering questions: which GPU, which precision, how much storage, which parallelism, which kernels. Then they wire it into their application and find out whether it actually serves their use case.

With our AI research team helping, that process took 15 to 20 working days. Almost a month per model. And we were doing it again and again for different customers.

So we asked a simple question: since we're doing this work anyway, can we do it once per model and let the customer skip it? That's Managed Endpoints. Pick a model from the catalog, click deploy, get a URL. The catalog currently includes DeepSeek V4 Flash, Gemma 4 31B, GLM-5.2, GLM-5.3 and GLM-5.3 Flash. You can configure how it scales, or take the defaults.

Where the 20 days actually go

An engineering lead reading this will think: I can pip install vllm and have a model serving in an afternoon. For a model that's been out for a year, that's true.

For a model that came out in the last 90 days, it isn't. The serving stack is still moving underneath you. Some models aren't supported in the latest release; you need a specific branch or a specific docker image. The recipe changes week to week. Getting a good tokens-per-second number out of it is its own research project, with hundreds of configuration knobs.

DeepSeek V4 Flash is a good example, and it's in our catalog. The vLLM docs tell you to follow a dedicated recipe rather than a bare pip install, because the model needs sparse attention backends and FP8 KV cache kernels. At one point the development branch was broken for it entirely. It failed to load on RTX PRO 6000 on the nightly image. As of last month it still doesn't run on A100s at all. The ROCm release notes claimed support and the image didn't come up on MI350X.

None of that is a criticism of vLLM. It's what "new model" means in 2026. Every one of those problems is a day or two for a team that has never hit it before.

What "managed" means, concretely

For every model in the catalog, our team pins the GPU, pins the image, picks the precision, sets the parallelism, and chooses the kernels. When we find an optimization that makes a model faster, we roll it into the image. You don't get a form with 200 fields; you get the configuration we would run ourselves.

On precision, we don't get clever. We serve each model at the precision its authors released and recommend, which is also what the major providers serve. GLM 5.2 ships an official FP8 checkpoint, so that's what runs. V4 Flash is served in mixed 4-bit. Then we check the result on a real benchmark before it ships.

Here's what that looked like for GLM 5.2. Our team burned over 1,000 GPU-hours and more than $5,000 characterizing the model on a single 8×H200 node. On Terminal-Bench 2.1, 89 real terminal tasks, GLM 5.2 on our platform scored 80.9%. Z.ai reports 81.0%. Artificial Analysis measured 77.9%. Three independent runs, all inside the noise window, so we reproduced the official number on our own platform.

That same work turned up the kind of thing you only find by running it. SGLang's default routing deals each request to the next GPU in line, blind to where a conversation's cache lives. Measured on our own traffic, 23% of all input tokens were being recomputed for no reason. The fix was one request header that pins each conversation to a GPU. Cache hits went from 64.7% to 92.1%, and median time-to-first-token halved. A team standing up their first endpoint would never see that. It's just baked into the image now.

The one thing we tell enterprises to do differently

Serverless products advertise scale-to-zero. For large models in production, we tell customers not to use it. Keep at least one worker warm.

Two reasons. First, availability. It's very hard to guarantee an H200 is free at 3am on a Tuesday, because demand is high and inventory is finite. Second, load time. With our current optimizations, GLM 5.2 takes around four to five minutes to load. We're working on bringing that down, but it's the current state.

Most enterprises are running the endpoint as a shared internal service. Multiple teams, different use cases. One team's cold start becomes everyone's outage. So for anything critical on a large model, one warm worker is the floor. If the workload isn't critical, or the model is small, scale to zero and don't worry about it.

What it costs

You pay per GPU-minute of worker runtime. If GLM 5.2 needs an 8×H200 node and it runs for 30 minutes, you're billed for 30 minutes. Our H200 is $3.99 per GPU-hour on demand, and committed pricing is available if you want to reserve capacity. Right now there is no premium for the managed layer. You pay the same rate you'd pay for the bare GPU. That will probably change in the coming months, so it's a good time to get in.

Let's say you keep one 8×H200 worker warm for a month. That's roughly $23,300 on demand, before any commitment discount. The same eight H200s on Together AI's cluster tier are $5.99 per GPU-hour, about $35,000. On Fireworks they're $7.00 per GPU-hour today, about $41,000, and Fireworks has announced an increase to $8.00 on September 1, which takes it to about $47,000.

Now the capacity side, based on our own experiments on that node at a floor of 20 tokens per second per user. (The full methodology and results are in our DataHack Summit 2026 talk.) One 8×H200 node holds 80 developers working at a realistic pace, or 38 agents running nonstop. Since a developer has an AI session running 25 to 50% of the workday, one node covers something like 160 to 320 people. On demand that works out to roughly $75 to $145 per developer per month, less on a commitment, and the bill doesn't move when they use it more.

Who should not use this

Per-token APIs are the right answer in three cases. Your usage is low or unpredictable. You're still experimenting and want to swap between five models in an afternoon without configuring anything. Or the model is small and you're fine with scale-to-zero, in which case either option works and the meter is simpler.

A dedicated endpoint wins when three things line up. You have sustained load: as a rough marker, somewhere around 15 always-on agents or 80 concurrent developers on a large model. You need latency and cost that don't change with usage. And you have data constraints. That last one is underrated. If your prompts can't leave a particular region, or a particular boundary, a per-token API somewhere else is off the table regardless of volume, and the endpoint is the only option that keeps the data where it has to be.

The trade-off is warm capacity against response time. If you need real-time responses on a large model, you keep a worker warm and pay for it whether or not it's busy, so it should be busy. If you don't, scale to zero and you pay nothing while it sits idle. What you shouldn't do is pay for a warm worker and then send it ten requests a day. That's the one case where the meter is the better deal.

Why this matters now

In July, Satya Nadella wrote that enterprises using proprietary AI are "paying twice": once for usage, and again by handing over their business knowledge through prompts and corrections. Later that month he told CNN that companies relying wholly on one proprietary lab "may not survive," and that the fix is to keep your harness separate from the model so "any one model can go away" and you stay in control. Microsoft, Palantir, NVIDIA, Meta and IBM signed a joint statement backing open-weight models.

He didn't say "use open models." He didn't have to. The gap between that advice and doing it has always been the 20 days. Open models are good enough now. Picking the right one for your work is still your call. Serving it well is the part that ate the 20 days, and that's what we built Managed Endpoints to shorten.

What's next

More models, and more modalities: speech-to-text, text-to-speech, text-to-video. A B200 cluster is coming in the next few months and will be available through Managed Endpoints when it lands. And later this year we plan to ship Token Factory, a per-token service on the same infrastructure, so a team that's too small for a dedicated endpoint today can start on the meter and move to a dedicated endpoint when the load justifies it.

You don't need to talk to anyone to deploy. Sign in, open the catalog, pick a model, and you'll have an endpoint. If you want help sizing a node for your team or working through data residency, talk to us.