vLLM Production Monitoring: A Practical Stack Guide for AI Engineers

vLLM has become the de facto open-source inference engine for serving large language models in production. Its PagedAttention mechanism delivers 2-5x throughput improvements over naive Hugging Face Transformers by managing the KV cache memory more efficiently. But that performance gain comes with new operational complexity: GPU memory pressure, KV cache hit rates, prefill/decoding phase imbalances, and speculative decoding behavior all require specialized monitoring that traditional DevOps tooling misses.

This guide walks you through building a production monitoring stack for vLLM covering what metrics matter, which open-source tools to use, and how to wire everything together.

Why vLLM Needs Its Own Monitoring Stack

Traditional API services expose request rate, error rate, and latency percentiles. vLLM is different. The core performance gains come from memory management decisions made inside the inference loop decisions that affect throughput by 2-10x depending on your workload.

The key things you need to monitor that standard HTTP observability will not catch:

• GPU memory utilization: vLLM pre-allocates a KV cache based on gpu_memory_utilization. If this is set too high, you will get OOM kills. Too low, you are wasting expensive GPU memory.
• KV cache hit rate: vLLM cache is your primary throughput lever. A low cache hit rate means you are recomputing tokens instead of serving from memory.
• Prefill vs decode throughput imbalance: If prefill is bottlenecking, you need different optimizations than if decode is the constraint.
• Number of ongoing sequences: vLLM block manager tracks active sequences. Understanding batch composition helps you tune max_num_seqs.

If you are running vLLM through Ray Serve (via vllm.entrypoint.api or serve run), you also have the Ray dashboard and Ray metrics to contend with.

The Metrics That Actually Matter

Throughput Metrics

• vllm:num_generation_tokens_total: Total tokens generated. Monitor rate.
• vllm:num_prefill_tokens_total: Total prefill tokens processed. Ratio to decode tracks workload shape.
• vllm:scheduler_running_steps: Number of active generation steps. Shows actual parallelism.

Memory Metrics

• vllm:gpu_cache_usage: KV cache memory used over allocated. 85-95% healthy, above 98% OOM risk.
• vllm:gpu_cache_usage_utilization: Normalized 0-1 scale. Primary tuning knob.
• vllm:num_mixed_chunked_prefill: Prefill batches mixed with decode. Indicates memory pressure.

Latency Metrics

• vllm:e2e_request_latency_seconds: Wall clock time from request to last token. p99 under 10s for most apps.
• vllm:time_to_first_token_seconds: TTFT, critical for streaming UX. p99 under 1 second.
• vllm:time_per_output_token_seconds: TPOT, inter-token latency. p99 under 100ms.

Setting Up Prometheus + Grafana for vLLM

Step 1: Enable vLLM Metrics

vLLM exposes metrics via a Prometheus endpoint at /metrics when you start the server. Make sure you are running with --enable-metrics (enabled by default in recent versions):

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --enable-metrics

The server will expose metrics at http://your-host:8000/metrics.

Step 2: Set Up Prometheus to Scrape vLLM

Add a scrape_config to your prometheus.yml:

scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: [localhost:8000]
    metrics_path: /metrics
    scrape_interval: 10s

If you are running behind a reverse proxy or on Kubernetes, adjust the target accordingly.

Step 3: Grafana Dashboard

Here is a starting dashboard JSON for Grafana. Import via + Import and paste the JSON:

GPU Cache Utilization Gauge

• Metric: vllm:gpu_cache_usage_utilization
• Type: Gauge
• Thresholds: green (0-0.85), yellow (0.85-0.95), red (0.95-1.0)

Tokens Generated per Minute

• Metric: rate(vllm:num_generation_tokens_total[5m])
• Type: Time series graph
• Shows actual throughput in tokens per second

Request Latency Percentiles (p50/p95/p99)

• Metric: histogram_quantile(0.50, rate(vllm:e2e_request_latency_seconds_bucket[5m]))
• Same for p95 and p99. Overlay all three on one graph.

Time to First Token (TTFT) p99

• Metric: histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
• Alert threshold: over 1 second for most models is noticeable UX degradation.

Scheduler Running Steps

• Metric: vllm:scheduler_running_steps
• Shows actual GPU parallelism. If this frequently drops to 1-2 while the queue is non-empty, you have a scheduling bottleneck.

Ray Dashboard: If You Are Running via Ray Serve

If you are using ray serve run or the vllm.entrypoint.api with Ray, you get the Ray dashboard for free:

ray start --head
# Access at http://localhost:8265

ray metrics --dashboard-address=localhost:8265

Ray metrics include:

• ray_num_task_executions: task throughput
• ray_resource_usage: CPU/GPU/memory per actor
• ray_get_task_latency: end-to-end task latency

Combined with vLLM native metrics, this gives you a full picture from request arrival to token delivery.

Common Issues and Detection Patterns

Issue 1: OOM Kills from Overallocated KV Cache

Symptom: vLLM container gets OOM-killed. nvidia-smi shows GPU memory at 99% right before crash.

Detection: Alert when vllm:gpu_cache_usage_utilization exceeds 0.95 for more than 5 minutes.

Fix: Reduce --gpu-memory-utilization by 5-10% increments. If you are below 0.80 and still OOMing, reduce max_model_len.

Issue 2: Thrashing Due to Too Many Concurrent Sequences

Symptom: Throughput drops suddenly. Scheduler running steps fluctuates wildly. Latency spikes.

Detection: If vllm:scheduler_running_steps frequently drops to 1-2 while the queue is non-empty, you have a scheduling bottleneck.

Fix: Lower --max-num-seqs (default 32). Fewer, larger batches often outperform many small sequences.

Issue 3: Low Cache Hit Rate

Symptom: Tokens per second is lower than expected for your hardware.

Detection: vLLM exposes KV cache metrics in recent versions including vllm:kv_cache_prefix_hit_rate if available.

Fix: Increase average prompt similarity (batch similar requests together), or serve with a larger gpu_memory_utilization to fit more sequences in cache.

Issue 4: Prefill Bottleneck

Symptom: TTFT is high but TPOT is normal. The model is slow to start generating.

Detection: Compare num_prefill_tokens_total rate against num_generation_tokens_total rate. If prefill rate is much higher than decode rate, you are prefill-bound.

Fix: Use continuous batching (vLLM default). For very long prompts, consider prompt caching or splitting across smaller chunks.

Cost Optimization: Getting More Tokens Per Dollar

vLLM efficiency directly translates to cloud spend. Here is how to optimize:

1. Tune gpu_memory_utilization with profiling

Profile your specific model to find the optimal setting. Reduce in 5% steps until OOM-free with headroom.

2. Use fp8 quantization for inference

Requires H100/A100 fp8 support. Delivers approximately 40% memory reduction with minimal accuracy loss.

3. Enable speculative decoding for lower latency workloads

Uses a smaller draft model to predict tokens, accepting speed-latency tradeoffs for batch inference scenarios.

4. Monitor tokens per GPU-second as your primary cost metric

Calculate as: tokens generated per hour divided by (number of GPUs times GPU cost per hour). This gives you cost per million tokens generated. If you are sizing a deployment across H100, A100, and L40S SKUs and want a back-of-envelope number before the benchmark runs, the LLM API cost calculator gives you a token-throughput-to-monthly-bill estimate in under a minute using current cloud pricing.

OpenTelemetry Integration

For enterprise environments, vLLM supports OpenTelemetry traces:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("vllm_generate") as span:
    outputs = llm.generate(prompts)
    span.set_attribute("num_tokens", len(outputs[0].outputs[0].token_ids))

Export to Jaeger, Datadog, or any OTLP-compatible backend. The same monitoring surface (Prometheus scrape + Grafana dashboards + OTel traces) keeps working regardless of whether the request took the native vLLM path or the transformers modeling backend path discussed below.

vLLM v0.25 Goodput Tuning (July 2026)

vLLM v0.25 makes Model Runner V2 the dense-model default and removes PagedAttention, so raw tokens per second is a weaker release KPI. Track goodput: requests per second that meet a latency contract. Sweep gpu_memory_utilization, max_num_batched_tokens, and max_num_seqs with pinned model, GPU, driver, workload, and SLO thresholds. Run chatbot, reasoning, and agentic workloads separately, publish a compliant-request counter to Prometheus, and alert on the goodput-to-throughput ratio.

Transformers vLLM Modeling Backend (July 2026)

On July 8, 2026, Hugging Face's Harry Mellor and Lysandre Jolidon published results from the new --model-impl transformers flag in vLLM. The result is that the transformers modeling backend now matches or exceeds native vLLM throughput on three reference workloads:

• Qwen3-4B dense: transformers backend equal within noise to native vLLM at single-node FP8
• Qwen3-32B TP=2: transformers backend slightly above native vLLM on token throughput at TP=2 across 2xH100
• Qwen3-235B-A22B-FP8 MoE DP+EP: transformers backend matches native vLLM at FP8 across 8xH100 with data-parallel + expert-parallel routing

This is the single biggest vLLM-side change since v0.19 (April 14, 2026). Before, anyone shipping a custom model had to port the modeling code into vLLM's internal backend to get PagedAttention, continuous batching, and speculative decoding. The --model-impl transformers flag removes that constraint: you can run unmodified Hugging Face transformers models on vLLM's serving stack and keep the production-grade scheduling and KV cache management.

When to choose which backend

• Native vLLM (the default): still the right call for legacy production stacks already tuned on it. Most production guides, dashboards, and incident playbooks written before July 2026 assume native vLLM. Switching has real cost: the metrics names, the OTel attribute names, and the error surfaces change.
• transformers backend: the right call for new models, models with custom modeling code (custom attention, custom MoE routing, custom linear-attention hybrids), and any team whose bottleneck is porting modeling code rather than serving scale. Custom attention implementations like linear attention, state-space models, or sliding-window attention can now ride vLLM without a port.

One-command migration cookbook

For teams already running vLLM, the migration is one flag plus one upgrade:

# Upgrade vLLM (the --model-impl transformers flag landed in v0.20.x)
uv pip install --upgrade vllm --torch-backend auto

# Smoke-test the same model with the transformers backend
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-4B \
  --model-impl transformers \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --enable-metrics

# Verify the load succeeded
curl -s http://localhost:8000/metrics | grep vllm:gpu_cache_usage_utilization

Linear-attention exception: some attention variants (linear-attention hybrids, RWKV-style state-space layers) still require the transformers backend because vLLM's native kernels target standard multi-head attention. If you are running a hybrid model, you are already on the transformers path by necessity; the new flag just makes that the supported, performant choice rather than a workaround.

Monitoring implications

The monitoring surface is identical between the two backends. The same DCGM exporter for GPU power and utilization, the same Prometheus scrape of /metrics, the same Grafana dashboards built around vllm:gpu_cache_usage_utilization and the latency histograms, and the same OpenTelemetry gen_ai.* spans (where supported) all work without modification. The only change is in the modeling code path, not in the inference loop or in what gets exposed.

What does change operationally:

• First-token latency variance increases slightly on the transformers backend for models that weren't ported (typically 5-15% higher p99 TTFT in the first 24 hours after switch). Plan a burn-in window.
• Out-of-memory patterns differ: native vLLM OOMs on KV cache exhaustion; transformers backend can OOM on activation memory for models with custom ops that aren't pre-fused. Watch vllm:gpu_cache_usage_utilization and DCGM_FI_DEV_FB_USED together.
• Speculative decoding (--speculative-model) still requires the native vLLM backend as of v0.20.x. If you rely on speculative decoding for latency, do not switch yet.

The Monitoring Stack at a Glance

The architecture that works:

• vLLM Server: exposes /metrics endpoint with native Prometheus format for GPU utilization, cache hit rate, latency histograms
• Prometheus: scrapes vLLM every 10s, stores time series, fires alerts on OOM risk and latency spikes
• Grafana: dashboards for KV cache utilization, throughput, latency percentiles, and token efficiency

If you are running vLLM in production today and not monitoring these metrics, start with two panels: GPU cache utilization (catch OOM risk before it kills your service) and request latency p99 (track user experience degradation). From there, add throughput efficiency metrics to correlate GPU spend with business value.

Next Steps

If you are running vLLM in production today and not monitoring these metrics, start with two panels:

1. GPU cache utilization: catch OOM risk before it kills your service
2. Request latency p99: track user experience degradation

From there, add throughput efficiency metrics to correlate GPU spend with business value.