Inference
Who serves it, and how fast
The same weights behave very differently depending on who runs them. These figures are medians across every provider serving a given model.
Providers tracked
81
Model–provider pairs
1,052
Fastest model
6,866 t/s
Apply 3
Lowest latency
64ms
Llama 3.1 8B Instruct
Serving performance
Speed and latency
Output speed decides how long a long answer takes. Latency decides how long the silence before it feels. They are not the same problem.
Output speed
Median tokens per second · higher is better
- Apply 3Relace6,866 t/s
- V3 LargeMorph3,208 t/s
- V3 FastMorph3,005 t/s
- gpt-oss-120bOpenAI641 t/s
- gpt-oss-safeguard-20bOpenAI537 t/s
- 461 t/s
- gpt-oss-20bOpenAI410 t/s
- Qwen3.8 27BQwen409 t/s
- 406 t/s
- GLM 5.2Z.ai368 t/s
- Schematron V2 SmallInference Net324 t/s
- Mercury 2Inception300 t/s
View as table
| Model | Tokens/s |
|---|---|
| Apply 3 (Relace) | 6866 |
| V3 Large (Morph) | 3208 |
| V3 Fast (Morph) | 3005 |
| gpt-oss-120b (OpenAI) | 641 |
| gpt-oss-safeguard-20b (OpenAI) | 537 |
| Llama 3.1 8B Instruct (Meta) | 461 |
| gpt-oss-20b (OpenAI) | 410 |
| Qwen3.8 27B (Qwen) | 409 |
| Grok 4.20 Multi-Agent (xAI) | 406 |
| GLM 5.2 (Z.ai) | 368 |
| Schematron V2 Small (Inference Net) | 324 |
| Mercury 2 (Inception) | 300 |
Time to first token
Median seconds · lower is better
- 64ms
- Gemma 4 31BGoogle89ms
- Phi 4Microsoft98ms
- Granite 4.1 8BIBM Granite118ms
- Llama 3 8B LunarisSao10K143ms
- 151ms
- 168ms
- gpt-oss-safeguard-20bOpenAI177ms
- Nemotron 3.5 LightningNVIDIA194ms
- Mistral Small 4Mistral AI194ms
- 212ms
- Schematron V2 SmallInference Net217ms
Reasoning models spend this window thinking, so a slow first token is not always a slow provider.
View as table
| Model | Latency |
|---|---|
| Llama 3.1 8B Instruct (Meta) | 64ms |
| Gemma 4 31B (Google) | 89ms |
| Phi 4 (Microsoft) | 98ms |
| Granite 4.1 8B (IBM Granite) | 118ms |
| Llama 3 8B Lunaris (Sao10K) | 143ms |
| Llama 3.3 70B Instruct (Meta) | 151ms |
| Nemotron 3.5 Content Safety (NVIDIA) | 168ms |
| gpt-oss-safeguard-20b (OpenAI) | 177ms |
| Nemotron 3.5 Lightning (NVIDIA) | 194ms |
| Mistral Small 4 (Mistral AI) | 194ms |
| Llama 3.2 3B Instruct (Meta) | 212ms |
| Schematron V2 Small (Inference Net) | 217ms |
Trade-off
Fast to start, or fast to finish
The top-right corner is what you want: a first token quickly, then tokens quickly after that. Very little sits there.
Output speed vs latency
Log latency axis, inverted so lower latency sits right
- Proprietary
- Open weights
Labelled points are the Pareto frontier — nothing in the index both starts sooner and streams faster.
View as table
| Model | Tokens/s | Latency |
|---|---|---|
| Apply 3 (Relace) | 6866 | 679ms |
| V3 Large (Morph) | 3208 | 323ms |
| V3 Fast (Morph) | 3005 | 570ms |
| gpt-oss-120b (OpenAI) | 641 | 250ms |
| gpt-oss-safeguard-20b (OpenAI) | 537 | 177ms |
| Llama 3.1 8B Instruct (Meta) | 461 | 64ms |
| gpt-oss-20b (OpenAI) | 410 | 225ms |
| Qwen3.8 27B (Qwen) | 409 | 273ms |
| Grok 4.20 Multi-Agent (xAI) | 406 | 3.86s |
| GLM 5.2 (Z.ai) | 368 | 233ms |
| Schematron V2 Small (Inference Net) | 324 | 217ms |
| Mercury 2 (Inception) | 300 | 1.14s |
| V4.1 Flash (DeepSeek) | 296 | 306ms |
| Nova Micro 1.0 (Amazon) | 270 | 298ms |
| Extract (Scaledown) | 256 | 1.07s |
| Nemotron 3.5 Content Safety (NVIDIA) | 255 | 168ms |
| M2.7 (MiniMax) | 252 | 324ms |
| Trinity Large Thinking (Arcee AI) | 246 | 714ms |
| Gemini 3.6 Flash (Google) | 242 | 762ms |
| Nemotron 3.5 Lightning (NVIDIA) | 226 | 194ms |
| V4 Flash 0731 (DeepSeek) | 223 | 422ms |
| Gemini 2.5 Flash Lite (Google) | 218 | 424ms |
| M2.5 (MiniMax) | 211 | 221ms |
| LFM2.5-2.6B (free) (Liquid) | 203 | 303ms |
| Nemotron 3 Nano 30B A3B (NVIDIA) | 201 | 362ms |
Providers
Who carries what
Catalogue size, and the median speed, latency and price of the models each provider serves.
Provider catalogue size
Models served · providers carrying at least three
- 70
- 59
- 58
- 57
- 47
- 45
- 41
- 40
- 39
- 38
- 26
- 26
- 25
- 22
A large catalogue usually means an open-weights host; a small one usually means a first-party API.
View as table
| Provider | Models | Median speed | Median latency | Median price / 1M |
|---|---|---|---|---|
| DeepInfra | 70 | 90 t/s | 450ms | $0.337 |
| Novita | 59 | 75 t/s | 668ms | $0.544 |
| OpenAI | 58 | 93 t/s | 1.43s | $4.19 |
| Azure | 57 | 93 t/s | 1.04s | $4 |
| 47 | 95 t/s | 902ms | $1.5 | |
| Alibaba | 45 | 72 t/s | 732ms | $0.75 |
| SiliconFlow | 41 | 90 t/s | 567ms | $0.394 |
| Parasail | 40 | 104 t/s | 402ms | $0.308 |
| Amazon Bedrock | 39 | 102 t/s | 930ms | $4 |
| Venice | 38 | 104 t/s | 416ms | $0.5 |
| Google AI Studio | 26 | 97 t/s | 2.07s | $1.31 |
| GMICloud | 26 | 98 t/s | 664ms | $0.514 |
| AtlasCloud | 25 | 100 t/s | 443ms | $0.525 |
| StreamLake | 22 | 95 t/s | 590ms | $0.534 |
DeepInfra
- Models served
- 70
- Median speed
- 90 t/s
- Median price / 1M
- $0.337
Novita
- Models served
- 59
- Median speed
- 75 t/s
- Median price / 1M
- $0.544
OpenAI
- Models served
- 58
- Median speed
- 93 t/s
- Median price / 1M
- $4.19
Azure
- Models served
- 57
- Median speed
- 93 t/s
- Median price / 1M
- $4
- Models served
- 47
- Median speed
- 95 t/s
- Median price / 1M
- $1.5
Alibaba
- Models served
- 45
- Median speed
- 72 t/s
- Median price / 1M
- $0.75
SiliconFlow
- Models served
- 41
- Median speed
- 90 t/s
- Median price / 1M
- $0.394
Parasail
- Models served
- 40
- Median speed
- 104 t/s
- Median price / 1M
- $0.308
Amazon Bedrock
- Models served
- 39
- Median speed
- 102 t/s
- Median price / 1M
- $4
Venice
- Models served
- 38
- Median speed
- 104 t/s
- Median price / 1M
- $0.5
Google AI Studio
- Models served
- 26
- Median speed
- 97 t/s
- Median price / 1M
- $1.31
GMICloud
- Models served
- 26
- Median speed
- 98 t/s
- Median price / 1M
- $0.514