Inference
Who serves it, and how fast
The same weights behave very differently depending on who runs them. These figures are medians across every provider serving a given model.
Providers tracked
72
Model–provider pairs
875
Fastest model
3,502 t/s
V3 Fast
Lowest latency
118ms
Granite 4.1 8B
Serving performance
Speed and latency
Output speed decides how long a long answer takes. Latency decides how long the silence before it feels. They are not the same problem.
Output speed
Median tokens per second · higher is better
- V3 FastMorph3,502 t/s
- V3 LargeMorph3,129 t/s
- Apply 3Relace3,080 t/s
- gpt-oss-120bOpenAI773 t/s
- GLM 4.7Z.ai502 t/s
- 443 t/s
- Qwen3 32BQwen423 t/s
- gpt-oss-safeguard-20bOpenAI422 t/s
- 337 t/s
- M2.7MiniMax307 t/s
- gpt-oss-20bOpenAI252 t/s
- GPT-5.6 Luna ProOpenAI242 t/s
View as table
| Model | Tokens/s |
|---|---|
| V3 Fast (Morph) | 3502 |
| V3 Large (Morph) | 3129 |
| Apply 3 (Relace) | 3080 |
| gpt-oss-120b (OpenAI) | 773 |
| GLM 4.7 (Z.ai) | 502 |
| Nano Banana 2 Lite (Gemini 3.1 Flash Lite Image) (Google) | 443 |
| Qwen3 32B (Qwen) | 423 |
| gpt-oss-safeguard-20b (OpenAI) | 422 |
| Grok 4.20 Multi-Agent (xAI) | 337 |
| M2.7 (MiniMax) | 307 |
| gpt-oss-20b (OpenAI) | 252 |
| GPT-5.6 Luna Pro (OpenAI) | 242 |
Time to first token
Median seconds · lower is better
- Granite 4.1 8BIBM Granite118ms
- 138ms
- Llama 3 8B LunarisSao10K158ms
- gpt-oss-120bOpenAI205ms
- Qwen3 32BQwen205ms
- Codestral 2508Mistral AI208ms
- 212ms
- MythoMax 13BGryphe222ms
- Hermes 4 70BNous Research229ms
- gpt-oss-safeguard-20bOpenAI237ms
- Command R7B (12-2024)Cohere238ms
- Step 3.7 FlashStepFun253ms
Reasoning models spend this window thinking, so a slow first token is not always a slow provider.
View as table
| Model | Latency |
|---|---|
| Granite 4.1 8B (IBM Granite) | 118ms |
| Llama Guard 4 12B (Meta) | 138ms |
| Llama 3 8B Lunaris (Sao10K) | 158ms |
| gpt-oss-120b (OpenAI) | 205ms |
| Qwen3 32B (Qwen) | 205ms |
| Codestral 2508 (Mistral AI) | 208ms |
| Llama 3.2 3B Instruct (Meta) | 212ms |
| MythoMax 13B (Gryphe) | 222ms |
| Hermes 4 70B (Nous Research) | 229ms |
| gpt-oss-safeguard-20b (OpenAI) | 237ms |
| Command R7B (12-2024) (Cohere) | 238ms |
| Step 3.7 Flash (StepFun) | 253ms |
Trade-off
Fast to start, or fast to finish
The top-right corner is what you want: a first token quickly, then tokens quickly after that. Very little sits there.
Output speed vs latency
Log latency axis, inverted so lower latency sits right
- Proprietary
- Open weights
Labelled points are the Pareto frontier — nothing in the index both starts sooner and streams faster.
View as table
| Model | Tokens/s | Latency |
|---|---|---|
| V3 Fast (Morph) | 3502 | 448ms |
| V3 Large (Morph) | 3129 | 476ms |
| Apply 3 (Relace) | 3080 | 1.01s |
| gpt-oss-120b (OpenAI) | 773 | 205ms |
| GLM 4.7 (Z.ai) | 502 | 311ms |
| Nano Banana 2 Lite (Gemini 3.1 Flash Lite Image) (Google) | 443 | 2.39s |
| Qwen3 32B (Qwen) | 423 | 205ms |
| gpt-oss-safeguard-20b (OpenAI) | 422 | 237ms |
| Grok 4.20 Multi-Agent (xAI) | 337 | 4.51s |
| M2.7 (MiniMax) | 307 | 304ms |
| gpt-oss-20b (OpenAI) | 252 | 259ms |
| GPT-5.6 Luna Pro (OpenAI) | 242 | 8.27s |
| Qwen3 Next 80B A3B Thinking (Qwen) | 242 | 313ms |
| Llama 3.3 70B Instruct (Meta) | 234 | 273ms |
| Gemma 4 31B (Google) | 232 | 517ms |
| Mercury 2 (Inception) | 228 | 572ms |
| Ling-3.0-flash (InclusionAI) | 211 | 293ms |
| Qwen3.5-35B-A3B (Qwen) | 203 | 419ms |
| Gemini 3.1 Flash Lite (Google) | 183 | 317ms |
| Gemini 3.5 Flash (Google) | 182 | 788ms |
| Nano Banana (Gemini 2.5 Flash Image) (Google) | 180 | 346ms |
| Qwen3.6 35B A3B (Qwen) | 177 | 254ms |
| Nova Micro 1.0 (Amazon) | 177 | 397ms |
| Nano Banana 2 (Gemini 3.1 Flash Image Preview) (Google) | 168 | 7.75s |
| GPT-5 Nano (OpenAI) | 168 | 1.31s |
Providers
Who carries what
Catalogue size, and the median speed, latency and price of the models each provider serves.
Provider catalogue size
Models served · providers carrying at least three
- 70
- 69
- 51
- 49
- 44
- 43
- 37
- 35
- 31
- 30
- 28
- 23
- 22
- 20
A large catalogue usually means an open-weights host; a small one usually means a first-party API.
View as table
| Provider | Models | Median speed | Median latency | Median price / 1M |
|---|---|---|---|---|
| Novita | 70 | 72 t/s | 512ms | $0.448 |
| DeepInfra | 69 | 73 t/s | 377ms | $0.37 |
| OpenAI | 51 | 56 t/s | 1.24s | $4.38 |
| Alibaba | 49 | 57 t/s | 560ms | $0.544 |
| Azure | 44 | 55 t/s | 1.14s | $4.19 |
| 43 | 102 t/s | 843ms | $1.13 | |
| Parasail | 37 | 72 t/s | 414ms | $0.29 |
| SiliconFlow | 35 | 76 t/s | 506ms | $0.355 |
| Venice | 31 | 84 t/s | 446ms | $0.475 |
| Amazon Bedrock | 30 | 74 t/s | 889ms | $3.13 |
| AtlasCloud | 28 | 81 t/s | 465ms | $0.525 |
| Google AI Studio | 23 | 106 t/s | 2.32s | $1.13 |
| StreamLake | 22 | 73 t/s | 514ms | $0.534 |
| Together | 20 | 89 t/s | 446ms | $0.534 |
Novita
- Models served
- 70
- Median speed
- 72 t/s
- Median price / 1M
- $0.448
DeepInfra
- Models served
- 69
- Median speed
- 73 t/s
- Median price / 1M
- $0.37
OpenAI
- Models served
- 51
- Median speed
- 56 t/s
- Median price / 1M
- $4.38
Alibaba
- Models served
- 49
- Median speed
- 57 t/s
- Median price / 1M
- $0.544
Azure
- Models served
- 44
- Median speed
- 55 t/s
- Median price / 1M
- $4.19
- Models served
- 43
- Median speed
- 102 t/s
- Median price / 1M
- $1.13
Parasail
- Models served
- 37
- Median speed
- 72 t/s
- Median price / 1M
- $0.29
SiliconFlow
- Models served
- 35
- Median speed
- 76 t/s
- Median price / 1M
- $0.355
Venice
- Models served
- 31
- Median speed
- 84 t/s
- Median price / 1M
- $0.475
Amazon Bedrock
- Models served
- 30
- Median speed
- 74 t/s
- Median price / 1M
- $3.13
AtlasCloud
- Models served
- 28
- Median speed
- 81 t/s
- Median price / 1M
- $0.525
Google AI Studio
- Models served
- 23
- Median speed
- 106 t/s
- Median price / 1M
- $1.13