Skip to content
llmwaves

Inference

Who serves it, and how fast

The same weights behave very differently depending on who runs them. These figures are medians across every provider serving a given model.

Providers tracked

81

Model–provider pairs

1,052

Fastest model

6,866 t/s

Apply 3

Lowest latency

64ms

Llama 3.1 8B Instruct

Serving performance

Speed and latency

Output speed decides how long a long answer takes. Latency decides how long the silence before it feels. They are not the same problem.

Output speed

Median tokens per second · higher is better

View as table
ModelTokens/s
Apply 3 (Relace)6866
V3 Large (Morph)3208
V3 Fast (Morph)3005
gpt-oss-120b (OpenAI)641
gpt-oss-safeguard-20b (OpenAI)537
Llama 3.1 8B Instruct (Meta)461
gpt-oss-20b (OpenAI)410
Qwen3.8 27B (Qwen)409
Grok 4.20 Multi-Agent (xAI)406
GLM 5.2 (Z.ai)368
Schematron V2 Small (Inference Net)324
Mercury 2 (Inception)300

Time to first token

Median seconds · lower is better

Reasoning models spend this window thinking, so a slow first token is not always a slow provider.

View as table
ModelLatency
Llama 3.1 8B Instruct (Meta)64ms
Gemma 4 31B (Google)89ms
Phi 4 (Microsoft)98ms
Granite 4.1 8B (IBM Granite)118ms
Llama 3 8B Lunaris (Sao10K)143ms
Llama 3.3 70B Instruct (Meta)151ms
Nemotron 3.5 Content Safety (NVIDIA)168ms
gpt-oss-safeguard-20b (OpenAI)177ms
Nemotron 3.5 Lightning (NVIDIA)194ms
Mistral Small 4 (Mistral AI)194ms
Llama 3.2 3B Instruct (Meta)212ms
Schematron V2 Small (Inference Net)217ms

Trade-off

Fast to start, or fast to finish

The top-right corner is what you want: a first token quickly, then tokens quickly after that. Very little sits there.

Output speed vs latency

Log latency axis, inverted so lower latency sits right

  • Proprietary
  • Open weights
010002000300040005000600010ms100ms1s10s100s1000sTime to first token, seconds · better →Output speed, tokens/sApply 3V3 Largegpt-oss-120bgpt-oss-safeguard-20bLlama 3.1 8B Instruct

Labelled points are the Pareto frontier — nothing in the index both starts sooner and streams faster.

View as table
ModelTokens/sLatency
Apply 3 (Relace)6866679ms
V3 Large (Morph)3208323ms
V3 Fast (Morph)3005570ms
gpt-oss-120b (OpenAI)641250ms
gpt-oss-safeguard-20b (OpenAI)537177ms
Llama 3.1 8B Instruct (Meta)46164ms
gpt-oss-20b (OpenAI)410225ms
Qwen3.8 27B (Qwen)409273ms
Grok 4.20 Multi-Agent (xAI)4063.86s
GLM 5.2 (Z.ai)368233ms
Schematron V2 Small (Inference Net)324217ms
Mercury 2 (Inception)3001.14s
V4.1 Flash (DeepSeek)296306ms
Nova Micro 1.0 (Amazon)270298ms
Extract (Scaledown)2561.07s
Nemotron 3.5 Content Safety (NVIDIA)255168ms
M2.7 (MiniMax)252324ms
Trinity Large Thinking (Arcee AI)246714ms
Gemini 3.6 Flash (Google)242762ms
Nemotron 3.5 Lightning (NVIDIA)226194ms
V4 Flash 0731 (DeepSeek)223422ms
Gemini 2.5 Flash Lite (Google)218424ms
M2.5 (MiniMax)211221ms
LFM2.5-2.6B (free) (Liquid)203303ms
Nemotron 3 Nano 30B A3B (NVIDIA)201362ms

Providers

Who carries what

Catalogue size, and the median speed, latency and price of the models each provider serves.

Provider catalogue size

Models served · providers carrying at least three

A large catalogue usually means an open-weights host; a small one usually means a first-party API.

View as table
ProviderModelsMedian speedMedian latencyMedian price / 1M
DeepInfra7090 t/s450ms$0.337
Novita5975 t/s668ms$0.544
OpenAI5893 t/s1.43s$4.19
Azure5793 t/s1.04s$4
Google4795 t/s902ms$1.5
Alibaba4572 t/s732ms$0.75
SiliconFlow4190 t/s567ms$0.394
Parasail40104 t/s402ms$0.308
Amazon Bedrock39102 t/s930ms$4
Venice38104 t/s416ms$0.5
Google AI Studio2697 t/s2.07s$1.31
GMICloud2698 t/s664ms$0.514
AtlasCloud25100 t/s443ms$0.525
StreamLake2295 t/s590ms$0.534