Serverless GPU inference.
Serve your model through an API with your choice of framework, GPU configuration and worker scaling.
Billed by the minute of worker runtime. Storage is billed separately.
Serving frameworks
Choose how your model runs.
Start with a supported framework and provide the model you want to serve.
Worker lifecycle
Set when workers start and stop.
Balance response time and idle cost with explicit worker limits and timeouts.
Keep capacity ready
Set a minimum worker count above zero for traffic that needs a warm model. Those workers are billed while they run, including time between requests.
Scale as requests arrive
Workers start as requests queue, within your maximum worker limit and available GPU capacity.
Stop idle workers
An idle timeout controls when workers stop. Set minimum workers to zero to allow the entire pool to stop; future requests may wait for a cold start.
Deployment settings
Control your worker pool.
Set GPU resources, concurrency and timeouts in the deployment form.
| Setting | Range | Purpose |
|---|---|---|
| GPU type | NVIDIA L4 · NVIDIA H100 · NVIDIA H200 · NVIDIA RTX Pro 6000 | One type per deployment |
| GPUs per worker | 1 · 2 · 4 · 8 | Size one worker to fit the model |
| Min workers | 0 to 100 | Set the warm-capacity floor |
| Max workers | 1 to 100 | The ceiling autoscaling can reach |
| Concurrent requests | 1 to 100 (Ollama: 4) | Per worker, before the queue grows |
| Idle timeout | 0s to 24h | How long a quiet worker stays warm |
| Wait time | 0s to 10m | How long a request waits for a worker before failing |
| File storage | 50 GB to 1 TB | Holds the model weights, billed around the clock |
Usage and billing
Understand the cost of a deployment.
GPU runtime and retained storage are separate parts of your bill.
GPU worker runtime
Each worker is billed by the minute while running. Check current GPU rates in the dashboard before deploying.
Worker disks and model storage
A running worker has a 50 GB root disk. File storage retains model weights and is billed at $0.00014/GB/hour even when the pool is stopped.
Logs and running cost
Inspect live logs, worker state and accrued costs in the dashboard to understand what the deployment is doing.
Choose your inference product
Your configuration or ours.
Choose based on who should maintain the model’s serving setup.
| Decision | Serverless | Managed Endpoints |
|---|---|---|
| Model selection | A model supported by your framework | A model in the managed catalog |
| Serving configuration | You choose the framework and GPU | Jarvislabs maintains the configuration |
| Best fit | Custom configuration and experimentation | A maintained model deployment |
Before you start
Common questions.
Practical details for choosing and using this product.
Which models can I serve?
Use a model supported by your chosen framework: vLLM, SGLang or Ollama. Enter a Hugging Face repository or Ollama tag as appropriate. Compatibility depends on the model, framework and GPU memory; arbitrary custom containers are not supported.
How is Serverless different from Managed Endpoints?
With Serverless, you choose the framework, model and GPU configuration. Managed Endpoints use catalog models with serving configurations maintained by Jarvislabs. Both let you configure worker scaling.
What happens after the deployment scales to zero?
A new request needs a worker to start and load the model. The request waits up to your configured wait time and fails if no worker becomes ready. Keep a minimum worker running when your application cannot tolerate that wait.
What am I charged when no requests arrive?
Running workers still incur charges. With minimum workers set to zero, the pool can stop after the idle timeout; file storage remains billed at $0.00014/GB/hour. Check the dashboard for current GPU rates.
Can I use the OpenAI SDK?
Yes. Use the endpoint URL and your Jarvislabs API token with the chat-completions API. Model features depend on the serving framework and the model you deploy.
Can I change a deployment after launch?
You can edit its name, idle timeout and wait time while it runs. Changing the model, GPU or worker configuration requires recreating the deployment.
Deploy your inference endpoint.
Choose a framework, select your model and configure the workers it needs.