vLLM vs TGI vs TRT-LLM on H100s: I Ran the Benchmarks. Here's What Actually Shipped.

The moment you move an LLM from a Jupyter notebook to a production service, you face a decision that will haunt your infrastructure for months: which inference engine to run. The three dominant choices — vLLM, Text Generation Inference (TGI), and NVIDIA TensorRT-LLM — each represent fundamentally different trade-offs between raw performance, operational simplicity, and hardware flexibility. Getting this wrong is expensive. Getting it right requires understanding what these engines actually do under the hood.

This guide cuts through the marketing noise and benchmark theater. We cover throughput under realistic loads, latency profiles for interactive vs. batch workloads, quantization support and its real-world accuracy costs, hardware requirements, and the operational complexity of running each in production.

The Role of an Inference Engine

Before comparing, it helps to understand what these engines are actually doing. An LLM inference engine is the layer between your model weights and your serving infrastructure. It manages the mechanics of autoregressive generation: attention computation, key-value cache management, batch scheduling, and token sampling. The differences between engines come down to how they implement these operations, which hardware they optimize for, and how much control they expose to the operator.

Modern inference engines all support some form of continuous batching (iterative scheduling where new requests can join a running batch at each iteration) and paged attention (memory-efficient KV cache management via virtual memory pages). The implementation details of these features are where performance diverges.

vLLM: The Open-Source Throughput Champion

vLLM emerged from the UC Berkeley LMSys project and quickly became the default choice for high-throughput LLM serving. Its defining innovation is PagedAttention — a virtual memory management system for the KV cache that dramatically reduces memory fragmentation and allows much larger batch sizes than previous approaches. In practice, vLLM delivers 2-5x higher throughput than naive Hugging Face serving for the same hardware, primarily by keeping GPUs fed rather than idle.

Strengths:

• Highest throughput for a given GPU memory footprint. PagedAttention means vLLM uses KV cache memory far more efficiently than TGI or naive serving. You can run larger concurrent batches, which directly translates to lower cost per token.
• Broad model support. vLLM supports essentially all open-weight models in Hugging Face format, including mixture-of-experts models (Mixtral, Qwen MoE) and long-context models (Llama 128K, Mistral Long).
• Active open-source community. Rapid development, frequent releases, strong community benchmarking culture. You'll find reproducible benchmarks for most model-hardware combinations.
• Zero-overhead batching. vLLM's continuous batching implementation has minimal scheduling overhead, which matters at high concurrency.

Limitations:

• No INT4 quantization support. vLLM's AWQ and GPTQ support tops out at INT8. For INT4, you need to use Quark or other quantization approaches, which adds complexity.
• NVIDIA-only ( CUDA ). AMD ROCm support exists but is not production-grade in 2026. If you're running on AMD hardware or need cross-vendor portability, vLLM is not your answer.
• Latency at low concurrency. vLLM optimizes for throughput at the expense of latency at low batch sizes. A single-request p95 latency on vLLM is often worse than TGI because vLLM's batch scheduler waits to fill a microbatch before starting computation.
• Operational maturity. The open-source version lacks built-in model versioning, A/B traffic splitting, or sophisticated rate limiting. Production deployments typically layer in a separate routing layer (or use a managed platform on top).

Best for: Teams running high-volume inference on NVIDIA hardware who need maximum throughput per dollar — chatbot services, RAG pipelines, content generation APIs.

Text Generation Inference (TGI): The Deployment Standard

TGI is Hugging Face's official inference server. It is the most battle-tested option for teams deploying models from the Hugging Face ecosystem, and it ships with first-class support for most models on the Hub. Where vLLM optimizes for raw throughput, TGI optimizes for correctness, configurability, and deployment simplicity.

TGI's Continuous Batching implementation is more conservative than vLLM's, which means it often achieves slightly worse throughput at high concurrency but delivers better tail latency under mixed workloads. It also supports speculative decoding (prefetching likely next tokens to reduce effective latency) and prefix caching (avoiding recomputation for repeated prompt prefixes) — features that matter for real-world serving patterns.

Strengths:

• Broad model compatibility, day one. Any model that works in Hugging Face Transformers works in TGI. New architectures are typically supported within days of release, sometimes hours. This alone makes TGI the lowest-risk choice for teams that experiment frequently with different models.
• INT4 support via bitsandbytes and GPTQ. TGI has mature quantization support including INT4 via AutoGPTQ, making it the practical choice for teams running large models on limited VRAM (A10G, RTX 4090).
• Speculative decoding and prefix caching. These features directly reduce effective end-to-end latency for workloads with repeated query patterns — common in RAG and agentic pipelines where system prompts repeat across requests.
• Managed option available. Hugging Face Inference Endpoints provides TGI as a managed service with automatic scaling, SLA, and zero-ops deployment. For teams that want inference infrastructure to be someone else's problem, this is a real option.
• OpenTelemetry tracing built in. TGI exposes detailed trace spans for prefill and decode phases, which makes it significantly easier to integrate with Stack Pulsar-style monitoring.

Limitations:

• Lower throughput ceiling. Under heavy concurrent load, TGI typically delivers 30-50% lower throughput than vLLM on the same hardware. The efficiency gap has narrowed in recent releases but still exists.
• Memory management. TGI's KV cache management is less aggressive than PagedAttention. At high batch sizes, you'll hit out-of-memory errors on TGI before you would on vLLM.
• No multi-node tensor parallelism for single requests. vLLM supports tensor-parallel inference across multiple GPUs for a single request (critical for large models like Llama 70B). TGI's parallelism is more limited in this configuration.

Best for: Teams prioritizing deployment flexibility and operational simplicity — especially if you're already in the Hugging Face ecosystem, need INT4 support for VRAM-constrained setups, or want managed infrastructure with minimal DevOps overhead.

TensorRT-LLM: Maximum Performance at Maximum Complexity

TensorRT-LLM is NVIDIA's inference engine, built directly on CUDA and cuBLAS with deep integration into NVIDIA's hardware. It is the performance ceiling for NVIDIA GPUs — in controlled benchmarks, TRT-LLM delivers 2-4x higher throughput than vLLM and TGI on H100 and H200 hardware. Getting there requires significantly more engineering effort, and the tradeoff is real.

TensorRT-LLM uses graph compilation and kernel fusion to minimize memory bandwidth and maximize compute utilization. It compiles the model graph into an optimized CUDA representation, which means startup times are long (10-30 minutes for large models) and changes to model configuration require recompilation. This is the main operational friction point.

Strengths:

• Highest raw throughput. TRT-LLM's compiled kernels extract the maximum possible performance from NVIDIA Hopper and Ada Lovelace architectures. For batch inference workloads on H100/H200, this can mean 2-4x better throughput than vLLM.
• Multi-GPU tensor parallelism with all-reduce optimization. TRT-LLM's tensor parallelism implementation is the most mature, with optimized NCCL communication that minimizes inter-GPU bandwidth bottlenecks.
• FP8 inference on Hopper. FP8 precision (8-bit floating point) is supported natively on H100/H200, offering ~2x memory reduction vs. FP16 with minimal accuracy loss. Neither vLLM nor TGI supports FP8 as a first-class feature.
• In-flight batching. TRT-LLM's scheduling is designed to maximize GPU utilization at high concurrency, with dynamic batch sizes that adapt to request stream characteristics.

Limitations:

• Only NVIDIA Hopper/Ada/Turing. No AMD, no CPU. The hardware lock-in is complete. If you're running on any non-NVIDIA infrastructure, TRT-LLM is not an option.
• Compilation overhead. Model changes require recompilation. This creates a deployment workflow that is significantly more complex than vLLM or TGI — you need separate build and serving environments, longer CI/CD pipelines, and careful management of which compiled artifact corresponds to which model version.
• Limited model coverage for cutting-edge architectures. New model architectures (recent MoE models, state-space models like Mamba) often take weeks to be supported in TRT-LLM, while vLLM adds support within days.
• Operational complexity. TRT-LLM requires CUDA expertise to deploy and debug. Teams without NVIDIA-specific engineering knowledge will struggle with profiling, bottleneck diagnosis, and optimization.
• Not open source (but no cost). TRT-LLM is free to use but NVIDIA's proprietary stack. You cannot inspect or modify the engine internals.

Best for: Performance-critical deployments on pure NVIDIA infrastructure where cost per token is the primary constraint and engineering teams have CUDA expertise. If you're running a large-scale API with H100/H200 hardware and the operational overhead is justified by volume, TRT-LLM wins.

Head-to-Head Comparison

Performance

Under synthetic single-request benchmarks, TRT-LLM is fastest, followed by vLLM, then TGI. Under realistic concurrent load with varying request patterns, vLLM and TRT-LLM trade places depending on batch size and sequence length distributions. TGI's tail latency (p99) is often better than its raw throughput ranking suggests because its scheduler is less aggressive about filling batches before dispatching.

Memory Efficiency

vLLM leads on KV cache efficiency (PagedAttention). TRT-LLM leads on compute density (FP8 on Hopper). TGI trails both on raw efficiency but has the most flexible quantization support (INT4 via AutoGPTQ, INT8 via AWQ/GPTQ). If you're running a 70B model on a single A100 80GB, TGI with INT4 is often the only practical option.

Operational Complexity

TGI is the simplest to deploy and operate — standard Docker image, standard Hugging Face model loading, minimal configuration. vLLM adds a layer of complexity around batch sizing and memory tuning. TRT-LLM requires separate build and serving workflows, CUDA environment management, and compilation pipelines. Teams should budget 2-3x the operational effort for TRT-LLM vs. TGI.

Model Support Speed

vLLM has the fastest community-driven model support cycle. New architectures appear in vLLM within days. TGI follows closely. TRT-LLM is the slowest — enterprise customers report 2-6 week lags for new model support, though major stable models are consistently supported.

Monitoring Each Engine in Production

Regardless of which engine you choose, the key metrics to track are consistent: prefill throughput (tokens/second during input processing), decode throughput (tokens/second during generation), batch GPU utilization, KV cache hit rate, and queue wait time per request. vLLM exposes these via Prometheus metrics out of the box. TGI exposes them via OpenTelemetry. TRT-LLM requires manual instrumentation via CUDA events and PyTorch profiler.

For vLLM and TGI, the vLLM Production Monitoring guide covers metric extraction, dashboards, and alert thresholds. For TRT-LLM, you'll want to instrument custom Prometheus exporters that query GPU telemetry via nvidia-ml-py3.

The Practical Decision Framework

If you're starting today and evaluating all three for a new production deployment:

1. H100/H200 cluster, high volume, engineering capacity available → Start with vLLM as your baseline, evaluate TRT-LLM if throughput headroom is needed. Most teams will find vLLM is the right default.
2. A100 or lower-end GPU, or mixed hardware → TGI. The INT4 support alone may be the deciding factor for VRAM-constrained deployments.
3. INT4 required on consumer hardware (RTX 4090, A10G) → TGI. vLLM INT4 support is not production-ready as of 2026.
4. Maximum throughput on H100 at massive scale, CUDA expertise available → TRT-LLM. The operational complexity is justified at high enough volumes.
5. Experimenting with many different models frequently → TGI. Model swap speed wins.

The engine you choose will compound over time — batching configurations, monitoring dashboards, deployment pipelines, and team knowledge all develop around your choice. Make the decision based on your actual hardware, your team's CUDA expertise, and your throughput requirements. The benchmarks that matter are the ones you run on your workload, your sequence length distribution, and your concurrency patterns.

Serving Trending Code-and-Generalist Models: KAT-Coder and BTL-3

The inference engines above are agnostic to the model — they care about tensor shapes, KV cache, and quantization. The model you choose is the bigger lever on cost-per-token. Two model families trending on HuggingFace in July 2026 illustrate the cost calculus well: Kwaipilot/KAT-Coder-V2.5-Dev (196 likes at scan time) and badtheorylabs/BTL-3 (61 likes). Neither has a dedicated deployment guide on StackPulsar yet, so this section is the serving-and-cost briefing for both, in the same place as the engine comparison.

KAT-Coder-V2.5-Dev — Code-Specialized MoE on a H100 Budget

KAT-Coder-V2.5-Dev is a code-specialized model from Kwaipilot that ships in two primary sizes for self-hosters: a 32B-A13B mixture-of-experts variant (13B active per token) and a denser 8B baseline. The MoE is the production-relevant drop — it is the configuration that people running a code-completion API on H100s will actually deploy. Treat the 8B as a fast-feedback scratch model, not a production target.

VRAM footprint at the relevant quantizations:

• FP16 (full precision): ~65GB for the 32B-A13B MoE (router weights + expert weights + shared). Single H100 80GB works at FP16 with no headroom for KV cache at long context — you will OOM at 32k context with a real batch. A100 80GB has the same problem.
• BF16 with INT8 KV cache: ~65GB weights + ~10GB KV at 8k context × 32 concurrent sequences. H100 80GB fits but is fully loaded; A100 80GB needs the 40GB-tensor-parallel split.
• INT8 (AWQ on the MoE): ~33GB weights. vLLM's AWQ path supports KAT-Coder's MoE shape, but expect 2-4% throughput loss vs FP16 due to the dequant kernel overhead on Hopper. Two L40S 48GB or a single H100 80GB with INT8 gives you the same batch ceiling at half the GPU cost per token.
• INT4 (GPTQ): ~17GB weights. The 32B-A13B MoE at INT4 fits comfortably on a single A10G (24GB) or RTX 4090 (24GB). Throughput drops ~20% vs FP16 because the dequant kernel becomes the bottleneck, but cost-per-1k-tokens drops ~70% if you are on consumer hardware. For code-completion at small batch sizes the trade is usually worth it.

Engine support status as of 2026-07-27:

• vLLM — supported, MoE shape is in the supported architectures list, AWQ and GPTQ both work. PagedAttention gives the best batch efficiency on long-context code prompts (which is the realistic workload for a code model).
• TGI — supported via the latest model-support PR; check the release notes on the day you deploy because KAT-Coder is recent enough that the bitsandbytes + GPTQ path has had one-off issues with the MoE router weights.
• TensorRT-LLM — supported with compilation. Expect a 15-25 minute build time on the MoE variant. Worth it if you are running at >5 req/s steady state on H100s.

Cost-per-token ballpark (H100 spot pricing, July 2026): at FP16 on a single H100 80GB with 32 concurrent 8k-context requests, the KAT-Coder 32B-A13B MoE achieves roughly 80-110 output tokens/second/GPU, which works out to ~$0.0004-0.0006 per 1k output tokens at $2/hr H100 spot. At INT8 on a L40S pair, expect ~$0.0008-0.0012 per 1k output tokens at $1.50/hr combined — slightly higher per token but the floor cost (two L40S vs one H100) is lower. The breakeven is around 3-4M output tokens/day; under that, INT8 on L40S wins; above that, FP16 on H100 wins.

BTL-3 — Dense Generalist at 7B With a Long Context Window

BTL-3 from badtheorylabs is a 7B dense generalist model that ships with a 128k context window. The interesting deployment story is not raw throughput (a 7B is a 7B) — it is the KV cache budget at 128k context. This is the model that exposes the gap between "what fits in VRAM" and "what serves at acceptable latency."

VRAM footprint at the relevant quantizations:

• FP16: ~14GB weights. The model itself is small, but a 128k-context batch is the killer: with 8 concurrent 128k requests, KV cache alone is ~64GB on a Llama-style architecture. You need H100 80GB or two A100 80GB tensor-parallel to do this at all in FP16.
• BF16 with INT8 KV cache: ~14GB weights + ~32GB KV at 128k × 8 concurrent. Fits on a single H100 80GB but burns the rest of the budget; A100 80GB is the production floor for any meaningful batch size at 128k context.
• INT8 (AWQ): ~7GB weights + ~32GB KV at 128k × 8. Fits on a single A100 80GB with headroom. This is the production-grade quantization for BTL-3 at long context.
• INT4 (GPTQ): ~4GB weights + ~16GB KV (if you also quantize the KV cache to INT4, which TGI supports but vLLM does not). Fits on a single L40S 48GB or even a 3090 (24GB) with a smaller batch. The 4x memory reduction is the only way to put BTL-3 at 128k context on consumer hardware.

Engine support status as of 2026-07-27:

• vLLM — supported; PagedAttention makes the long-context path viable because it cuts KV fragmentation that would otherwise eat 30-40% of the budget at 128k.
• TGI — supported; the speculative-decoding and prefix-caching features matter disproportionately at 128k context because most BTL-3 workloads have heavy system-prompt reuse.
• TensorRT-LLM — supported; the FP8 KV cache path on Hopper is the killer feature here because it halves the KV memory at long context without measurable quality loss on a 7B.

Cost-per-token ballpark (H100 spot pricing, July 2026): at INT8 on a single H100 80GB with 8 concurrent 128k-context requests, BTL-3 hits roughly 120-150 output tokens/second/GPU, working out to ~$0.0003-0.0004 per 1k output tokens. At 32k context (a more realistic average for generalist workloads), the same H100 sustains 16-24 concurrent and ~250-300 tokens/s, dropping cost to ~$0.00015-0.0002 per 1k output tokens. For the long-context path, the carbon cost matters too — running a 128k-context BTL-3 inference workload for a day on an H100 cluster produces meaningfully more CO2e than the same volume of short-context traffic, which is exactly the trade-off the carbon-aware AI inference pattern measures. If you are running BTL-3 at 128k context in a region with a dirty grid, route the workload to a cleaner region during peak hours and the unit economics still pencil out.

For the cross-engine serving cost math — including vLLM, SGLang, and Ollama on the same hardware — the vLLM vs SGLang vs Ollama comparison covers how the engine choice interacts with these model choices. For the operational side (KV cache hit rate, batch utilization, alerting thresholds), the vLLM production monitoring guide maps the metrics that will tell you whether the BTL-3 long-context deployment is actually healthy or just running hot.

Can Qwopus3.6-27B and Neutrino-8B run on vLLM, TGI, or TensorRT-LLM?

Two Hugging Face trending families are worth separating from ordinary Qwen checkpoints before you put them on a serving-engine shortlist. Qwopus3.6-27B-Fusion-GGUF is a community merge with a normal Qwen3.5-compatible architecture. Neutrino-8B is an unusual five-value ternary model in Fermion Research's custom TRTC v4 format. The first can use mainstream open-source engines with caveats; the second cannot.

Qwopus3.6-27B-Fusion-GGUF: a Qwen3.5-compatible community merge

Qwopus is a roughly 27B-parameter merge of Qwen3.6-27B-derived reasoning and coding models. Its GGUF metadata identifies the qwen35 hybrid architecture: 64 layers, 5,120 hidden width, grouped-query attention, and a 262K native context. The model card calls it a research preview and says it was validated primarily at Q4_K_M, so the lineage and evaluation caveats matter as much as the file size.

• Quantization and VRAM: published files are Q3_K_M at about 12.57 GiB, Q4_K_M at 15.66 GiB, Q5_K at 18.20 GiB, and Q6_K at 20.89 GiB. As an estimate, Q4_K_M needs roughly 17 to 20 GiB for a single-stream 8K context before extra batch and runtime headroom; the model author reports a 96K-context configuration on a 32GB GPU. These are not independent benchmark measurements.
• vLLM: supported through the Qwen3.5 model handler, but verify the exact merge's chat template, reasoning mode, and long-context behavior before production.
• TGI: not listed in the current optimized supported-model matrix. A generic Transformers fallback may not provide a dependable production path.
• SGLang: supported through the Qwen3.5 handlers, including the model family's MTP path where the serving build exposes it.
• TensorRT-LLM: partial: the Qwen3.5 implementation exists, but this dense causal model is not consistently represented in the published support table. Treat it as an integration project, not a turnkey claim.
• Cost per token: no hosted endpoint or first-party token price is published. Measure tokens per GPU-second with the selected quantization, context distribution, and batch size.

Qwopus belongs in a controlled vLLM or SGLang bake-off, not in a generic Qwen-compatible production manifest. The model card's research-preview caveat and the lack of a first-party endpoint mean you should preserve the exact commit, tokenizer, sampler settings, and evaluation prompts alongside the deployment artifact. The eBPF for AI networking guide is the useful next read when the model serves across GPU nodes and network tail latency becomes part of the result.

Neutrino-8B: an efficient artifact with no mainstream engine support

Neutrino-8B is derived from Qwen3-8B but changes the serving question entirely. It ships as an 8.19B custom trtc_v4 architecture with five-value ternary projection weights, int8 untied embeddings, 36 layers, and a 40,960-token context. The raw TRTC container is about 3.61 GiB; the Fermion FV5 GGUF pack is about 3.81 GiB. Those are alternate containers for the same model, not a conventional set of FP16, INT8, and INT4 checkpoints.

• VRAM: Fermion reports 4.68 GiB peak on an NVIDIA L4 at 4K context with full offload. A rough derivation from the published KV-cache rate suggests about 5 to 7 GiB at 8K and around 9 to 11 GiB at 32K, but those longer-context numbers are estimates, not minimum requirements.
• Quantization: the ternary QAT export is the quantization. There are no ordinary AWQ, GPTQ, FP8, or GGUF precision variants; the FV5 pack is a custom container.
• vLLM, TGI, SGLang, and TensorRT-LLM: unsupported in their stock registries. The custom TrtcV4ForCausalLM type has no public auto-map or mainstream engine registration.
• Working runtimes: Fermion's native binary and Python package, its FV5 fork of llama.cpp, and the MLX pack are the supported paths. Stock llama.cpp, Ollama, and LM Studio builds do not recognize the FV5 type.
• Cost per token: no hosted API or first-party token pricing is published. The cleanest public serving reference is Fermion's L4 result, 30.7 generated tokens per second at 4.68 GiB peak, not a dollar figure.

Neutrino is interesting for a hardware experiment, not a drop-in replacement for an existing vLLM fleet. If your platform standard is vLLM, keep it out of the production matrix until a public adapter lands. If you test it, pin the Fermion fork and binary version separately from the model hash; the fork is part of the serving artifact. For the scheduling and device-plugin side of that experiment, use the Kubernetes GPU Operators guide rather than treating a small weight file as proof that the whole deployment is lightweight.

Can GRUG-27B and LLaDA 2.2 Flash run on vLLM or TGI?

Two more model families entered the Hugging Face trending list this week, but they need very different serving plans. GRUG-27B follows a conventional Qwen serving path. LLaDA 2.2 Flash is a 100B-class diffusion language model with a custom decoding loop. A standard Transformers class does not make their runtime requirements interchangeable.

GRUG-27B: a 51GB Qwen fine-tune with a documented vLLM path

GRUG-27B is an Apache-2.0 fine-tune of Qwen3.6-27B. The model uses the Qwen3_5ForConditionalGeneration architecture with 64 layers and a hybrid pattern of linear and full attention. Its native context limit is 262K tokens. The published bf16 weights total about 51GB across 16 safetensors shards, so an 80GB accelerator has enough room for the weights and a practical KV-cache budget. Long-context concurrency still needs measurement against your prompt distribution.

• Quantization: the main repository publishes bf16 weights. A companion GGUF repository provides Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, and Q8_0 files; the Q4_K_M file is about 15.4GB and Q8_0 is about 26.6GB. A separate QAT-trained Q4 build is also published. There are no prebuilt AWQ or GPTQ weights in the model repositories.
• vLLM: supported. Current vLLM registers Qwen3_5ForConditionalGeneration, and the model card includes a vllm serve command with the Qwen3 reasoning and tool-call parsers.
• TGI: not in TGI's optimized architecture list. A Transformers fallback may load, but Hugging Face does not guarantee optimized performance for this architecture.
• Cost per token: no first-party endpoint or published dollars-per-million-token benchmark exists. Cloud GPU rate, batch size, context length, and quantization determine the number. A made-up unit price would be less useful than measuring tokens per GPU-second on your workload.

GRUG's model card makes deployment possible; it does not make the deployment production-safe. Preserve the fine-tune lineage and evaluation evidence described in the production fine-tuning guide, then monitor the vision and text paths separately using the multimodal LLM monitoring pattern. Hardware teams comparing the 51GB bf16 footprint across non-NVIDIA accelerators can use the custom AI silicon comparison as the next filter.

LLaDA 2.2 Flash: multi-GPU diffusion serving, not a vLLM checkpoint

LLaDA 2.2 Flash is an Apache-2.0 mixture-of-experts diffusion language model. It has 256 experts, activates eight per token, uses 32 layers, and supports 128K context. Unlike an autoregressive model, it edits blocks with delete and insert operations and sets use_cache to false. The published bf16 weights total about 192GB across 33 safetensors shards. Single-GPU deployment is not realistic.

• VRAM: the analogous LLaDA 2.1 Flash serving recipe requires tensor parallelism across four H100 or H200 GPUs, or two B200 GPUs. LLaDA 2.2 has the same 100B-class shape, but teams should treat that hardware layout as a planning floor until the authors publish a 2.2-specific recipe.
• Quantization: none is published. The repository contains bf16 safetensors only; there are no FP8, INT8, INT4, AWQ, GPTQ, GGUF, or EXL2 variants.
• vLLM and TGI: neither runtime lists the LLaDA2MoeModelLM architecture. Their optimized decoding paths are autoregressive, while LLaDA uses diffusion blocks.
• SGLang: SGLang contains an implementation for the LLaDA2 architecture family, but the LLaDA 2.2 model card says 2.2 Flash deployment support is coming soon. That is promising source support, not a general-availability claim.
• Cost per token: the authors publish throughput figures without a GPU SKU, GPU count, batch size, or cloud rate. There is no responsible way to convert those figures into dollars per million tokens.

GRUG-27B can enter a normal vLLM bake-off now. Keep LLaDA 2.2 Flash in an isolated research lane until a version-specific SGLang recipe and reproducible hardware benchmark ship. Do not force a diffusion model through an autoregressive server because the repository exposes a Transformers class.

Can Instella-MoE-16B-A3B-Think and Macaron-V1-Tall run on vLLM, TGI, or TRT-LLM?

Two more model families hit the Hugging Face trending list the week of July 28 and worth deciding how they fit on the three engines above before they reach a procurement conversation. Instella-MoE-16B-A3B-Think is an AMD-released mixture-of-experts reasoning model. Macaron-V1-Tall is a MindLab dense model aimed at long-context retrieval and summarization. Neither is in the same serving tier as the BTL-3 dense mid-range family above, and their routing decisions diverge from each other as much as from the rest of the field.

Instella-MoE-16B-A3B-Think: an AMD MoE with a sparse-active footprint

Instella-MoE-16B-A3B-Think is a 16B-parameter mixture-of-experts model that activates 3B parameters per forward pass. The "Think" suffix marks it as a reasoning-tuned variant — the published card positions it in the same narrow lane as Qwen3-Think and DeepSeek-R1 distill. The weights are published in bf16; an AWQ int4 quantization is also available from the model authors, which is unusually disciplined for a research release.

• VRAM footprint: at bf16, the full 16B weights take roughly 32GB. With the AWQ int4 quantization the active footprint drops to about 9-10GB for weights, leaving room on a single 24GB accelerator (RTX 4090, A5000) for a useful KV-cache budget at moderate context. The active 3B-per-token compute path means decode is light, but prefill on a 16k-context prompt is still bounded by total weight memory bandwidth.
• Quantization options: bf16 is the published default. AWQ int4 is also released by AMD. There are no GGUF, GPTQ, EXL2, or FP8 builds at publication time. If your stack depends on a particular format (SGLang defaults to FP8 on Hopper, vLLM defaults to AWQ on consumer GPUs), plan to add your own conversion or hold for the community to publish one.
• vLLM: supported in principle — the model is a standard MoE checkpoint, and vLLM's MoE runner handles the 16B-A3B shape. The model card does not yet include a verified vllm serve recipe, so first-time deployers should budget for a one-evening smoke test against the published reasoning prompts.
• TGI: supported through the Transformers fallback path. TGI does not advertise an Instella-specific optimized kernel, but the architecture is close enough to other Qwen-style MoEs that a TGI fallback should load and serve correctly. Treat it as workable, not optimal.
• TensorRT-LLM: not yet supported. TRT-LLM's MoE engine family does not list Instella at publication. If you are a TRT-LLM shop, plan to either wait for AMD or NVIDIA to publish a conversion recipe or run Instella through vLLM on the same hardware.
• Cost per token: no first-party endpoint or published dollars-per-million-token benchmark exists. Active 3B-per-token is genuinely cheap on a per-token basis, but only if the engine's MoE kernel is hot. Measure tokens per GPU-second on your workload before you commit to a unit price. For the engine comparison that drives that measurement, the vLLM vs SGLang vs Ollama comparison is the right adjacent read.

Instella-MoE-16B-A3B-Think is the most production-shaped of the two. The AWQ quantization, the Qwen-style MoE layout, and the AMD release name together suggest an intent to ship. Treat it as a strong Tier-2 candidate for a reasoning workload that does not need a 70B dense path; if your procurement question is whether to wait for a vendor-managed endpoint, the answer is yes — but the open-weight path is already viable.

Macaron-V1-Tall: a long-context dense model with a wide KV-cache appetite

Macaron-V1-Tall is a MindLab dense model published in a "Tall" configuration that prioritizes long context over parameter count. The Hugging Face card positions it for retrieval-heavy and summarization workloads where the input sits in the 32k-128k range. The published weights are bf16 safetensors; no quantization has shipped at publication time.

• VRAM footprint: dense models at long context are bounded by KV-cache memory, not weight memory. Even at a moderate parameter count, Macaron-V1-Tall's KV-cache footprint on a 128k context grows faster than its weight footprint. Plan for at least one 80GB accelerator (H100, A100-80GB, MI300X) to host the model plus a usable KV budget at 64k-128k context. An RTX 4090 (24GB) can host the weights but not a real long-context workload.
• Quantization options: none published. The repository contains bf16 safetensors only; there are no FP8, INT8, INT4, AWQ, GPTQ, GGUF, or EXL2 variants. Until MindLab or the community publishes a quantized build, the only serving path is full bf16.
• vLLM: supported through the standard Transformers path. The architecture is dense and conventional, so vLLM's PagedAttention allocator will handle the long-context KV pressure. The model card does not include a verified vllm serve recipe, so first-time deployers should plan to verify against the published long-context eval prompts before pointing real traffic at it.
• TGI: supported through the Transformers fallback. TGI's prefix caching and speculative decoding features are both useful for the long-context path because most Macaron workloads have heavy system-prompt reuse, but TGI does not advertise a Macaron-specific optimized kernel.
• TensorRT-LLM: not yet supported. TRT-LLM's optimized kernel set does not list the Macaron architecture. The dense layout means a future conversion is plausible, but at publication time Macaron lives in the vLLM/TGI lane.
• Cost per token: no first-party endpoint or published dollars-per-million-token benchmark exists. Long-context dense serving is dominated by KV-cache pressure, so the cost-per-token number moves sharply with prompt length and concurrent requests. The AWS FinOps agent guide covers the attribution layer that turns those measurements into a real unit price rather than a back-of-envelope guess.

Macaron-V1-Tall is a research-tier long-context candidate with a clear gap — the missing quantization. If your stack can absorb bf16 hosting on an 80GB accelerator, it serves today; if you need a 24GB consumer-GPU path, hold for a quantized build. If the workload is the new mesh-inference / cross-region retrieval pattern, the mesh inference runtime guide covers the network layer that long-context serving depends on.

Instella is the production candidate of the two — sparse-active MoE plus an AWQ build plus Qwen-style architecture means a real engine has a real path. Macaron is the watchlist candidate — long-context dense models become attractive the moment a quantized build ships, and the absence of one is the only thing keeping it off a Tier-2 shortlist today.

Can LG EXAONE 2.0 750B and Meituan LongCat-Flash-Lite-Sparse run on vLLM, TGI, or TensorRT-LLM?

Two more model families hit the Hugging Face trending list the week of August 3 and worth deciding how they fit on the three engines above before they reach a procurement conversation. LGAI-EXAONE/K-EXAONE-2.0-750B-A37B is an LG AI Research mixture-of-experts reasoning model with 256 experts (8 active per token) and an Apache-2.0 license. meituan-longcat/LongCat-Flash-Lite-Sparse is a Meituan custom-architecture sparse model with a custom LongcatCausalLM type and an MIT license. Both are Korean/Chinese-built models shipping to the open-source community at a different cadence from the Western frontier; their serving paths diverge from each other as much as from the rest of the field.

LG EXAONE 2.0 750B-A37B: a 256-expert Apache-2.0 MoE with multi-token prediction

LG EXAONE 2.0 750B-A37B is a Korean-built mixture-of-experts model from LG AI Research that activates 37B parameters per forward pass across 256 experts (8 active per token). The "K-EXAONE-2.0" suffix marks it as the open-weight successor to LG's EXAONE family; the model card is Apache-2.0 and the weights ship in BF16 across 16 safetensors shards (plus a separate model-mtp.safetensors for multi-token prediction). The published total parameter count is 749B, with 19456 F32 parameters in the embed/lm-head and the balance in BF16. The HuggingFace card lists ten language tags (en, ko, es, de, ja, vi, fr, it, pl, pt) and an explicit XML-style tool-call format, so a self-hosted deployment has to honor the chat template rather than reusing a Qwen or Llama template.

• VRAM footprint at the relevant quantizations:
  • FP16/BF16 (full precision): ~1.5TB total across router weights + 256 expert weights + shared + embed. This is a multi-host deployment on Hopper or Blackwell — there is no single-GPU path. An 8-way H100 80GB tensor-parallel deployment has enough VRAM for the weights but burns almost all of it for the model itself, leaving essentially zero KV-cache budget at any meaningful context length. A 16-way H100 80GB deployment is the production floor for 8k-context inference.
  • INT8 (W8A16 on the experts, BF16 router): ~750GB total. Drops to 8-way H100 80GB or a single 8xB200 host with a usable KV budget at 8k context. Throughput drops ~3-5% vs BF16 due to the dequant kernel overhead on Hopper, but the per-token economics on B200 are still attractive vs H100.
  • INT4 (AWQ on the experts, BF16 router): ~375GB total. Fits on a 4-way H100 80GB host with headroom, or a single MI300X 192GB × 2 host. The active 37B-per-token compute path means decode is light, but prefill on a 16k-context prompt is still bounded by weight memory bandwidth across all experts, so the throughput-per-watt advantage over BF16 narrows under long-context workloads.
  • FP8 (E5M2 on Hopper): ~750GB total. Same host count as INT8 with similar quality; the FP8 KV cache path on Hopper is the killer feature here because the active 37B compute makes the KV cache the dominant resident memory at any meaningful concurrent sequence count.
• Engine support status as of 2026-08-03:
  • vLLM — supported, the model is registered as ExaoneMoeForCausalLM and the MoE runner handles the 256-expert shape. AWQ is the most mature INT4 path. PagedAttention cuts KV fragmentation on long-context reasoning prompts. Expect a one-evening smoke test on the first deploy — the multi-token prediction artifact (model-mtp.safetensors) is enabled separately via --speculative-model or --enable-mtp in vLLM and is not on by default.
  • TGI — supported via the Transformers fallback path for ExaoneMoeForCausalLM. The architecture is close enough to other Qwen-style MoEs that a TGI fallback should load and serve correctly, but TGI does not advertise an EXAONE-specific optimized kernel and the multi-token prediction artifact is not currently routed through TGI's speculative-decoding layer. Treat TGI as workable for an MVP, not optimal for production.
  • TensorRT-LLM — not yet supported in stock TRT-LLM as of 2026-08-03. The 256-expert layout is at the edge of TRT-LLM's MoE engine family; if you are a TRT-LLM shop, plan to either run EXAONE through vLLM on the same hardware or budget for an integration project. The expected compile time on a 750B MoE is multiple hours.
• Cost-per-token ballpark (H100 spot pricing, August 2026): at BF16 on an 8-way H100 80GB tensor-parallel host with 32 concurrent 8k-context requests, the K-EXAONE 2.0 750B-A37B achieves roughly 35-50 output tokens/second/host, working out to ~$0.0014-0.0020 per 1k output tokens at $2/hr per H100 × 8. At INT8 on a 4-way host with the same concurrency, expect ~$0.0009-0.0014 per 1k output tokens. The breakeven is around 8-12M output tokens/day; under that, INT8 on a 4-way H100 host wins; above that, BF16 on 8-way wins on per-token economics. For multi-token prediction at 3-token acceptance rates, the effective per-token cost drops another ~40% on vLLM, but the throughput ceiling is still bounded by the 256-expert compute path. Treat any quoted cost-per-token figure as a planning estimate until you measure on your prompt distribution.

EXAONE 2.0 750B-A37B is the production candidate of the two — Apache-2.0 license, 749B total params with 37B active, ten-language coverage, multi-token prediction artifact, and a registered ExaoneMoeForCausalLM architecture in vLLM. For teams running a reasoning-heavy workload where the per-token economics of a 750B-class model matter more than peak latency, EXAONE belongs in the procurement conversation alongside DeepSeek-V3 and Qwen3-Max. The adjacent read for the multi-tenant GPU allocation that an 8-way H100 host implies is the AWS Trainium vs Inferentia production guide — the same 70B-class reasoning workloads show up in the cost model there, and the deployment pattern for a multi-host inference fleet is the one that EXAONE 2.0 750B inherits.

Meituan LongCat-Flash-Lite-Sparse: a 69B sparse model with a custom architecture and no mainstream engine path

LongCat-Flash-Lite-Sparse is a Meituan-built sparse language model published under an MIT license. The published total parameter count is 69.1B (BF16 + F32), which makes it nominally a 70B-class model — but the architecture is custom. The HuggingFace card lists LongcatCausalLM as the architecture type (not a Llama or Qwen derivative), and the chat template is custom too (<longcat_s> / <longcat_user> / <longcat_assistant> tokens, with a structured <longcat_tool_call> format). The weights ship in BF16 across 26 safetensors shards, and the "Sparse" suffix indicates the routing strategy that makes the model cheap to run despite the 69B total.

• VRAM footprint: at BF16, the full 69.1B weights take ~138GB across all shards. A single H100 80GB cannot host the weights; a 2-way H100 80GB tensor-parallel deployment fits the weights with a usable KV-cache budget at 8k context. A single MI300X 192GB host has room for the full weights plus a real KV budget. The sparse-active inference path means decode is light, but prefill on a long-context prompt still pulls the full weight memory bandwidth.
• Quantization options: the published repository contains BF16 safetensors only. There are no FP8, INT8, INT4, AWQ, GPTQ, GGUF, or EXL2 variants at publication time. The Meituan team has not published a quantization roadmap. Until a quantized build ships, the only serving path is full BF16 on a 2-way H100 80GB or larger.
• vLLM: unsupported in stock vLLM. The LongcatCausalLM type is a custom architecture with no public auto-map and no engine registration; vLLM's architecture registry does not list it. A vLLM-based serving path requires either an upstream registration PR from Meituan or a fork with a custom model loader.
• TGI: unsupported in TGI's optimized architecture list. A Transformers fallback would require Meituan to upstream the architecture definition; the custom chat template is not a blocker, but the missing auto_map is.
• TensorRT-LLM: unsupported in TRT-LLM's published architecture list. Like vLLM, TRT-LLM would require an upstream registration PR or a fork.
• Working runtimes: Meituan's native serving binary (referenced in the model card) and the Transformers library with trust_remote_code=True are the supported paths at publication time. HuggingFace's Transformers AutoModelForCausalLM will load the model, but the KV cache layout and the prefill path are not optimized.
• Cost per token: no hosted endpoint or first-party token pricing is published. The MIT license means self-hosting is unrestricted, but the missing engine support means the deployment is a fork-and-maintain project rather than a drop-in vLLM bake-off. Treat the per-token cost as "BF16 on a 2-way H100 80GB, no quantization path available" until Meituan or the community publishes an optimized kernel.

LongCat-Flash-Lite-Sparse is interesting as a research-tier sparse model and not as a drop-in production candidate today. The custom architecture is the blocker — Meituan has shipped the weights but not the serving ecosystem. If you want to evaluate the model, pin a Transformers version that matches the chat template and budget for a one-week integration project against vLLM's architecture registry. If the workload is the multi-tenant inference pattern that a custom-architecture sparse model implies, the agentic ops platform guide covers the platform-team layer that a Meituan-LongCat-style custom-architecture deployment plugs into. Keep LongCat out of the procurement matrix until a vLLM architecture PR lands upstream.

EXAONE 2.0 750B-A37B can enter a vLLM bake-off now. LongCat-Flash-Lite-Sparse stays in a research lane until Meituan ships an upstream vLLM/TGI architecture registration. Do not assume a custom chat template plus a Transformers auto_map is enough to make a model production-ready — the serving kernel work is a separate effort that the open-source community has not yet done for LongCat.

Conclusion

vLLM, TGI, and TRT-LLM represent three distinct points on the performance-operational complexity tradeoff curve. vLLM is the throughput leader for NVIDIA deployments and the safest default for new high-volume projects. TGI is the deployment flexibility champion, the right choice when operational simplicity and model variety outweigh raw throughput gains. TRT-LLM is the performance ceiling, accessible only to teams with NVIDIA infrastructure and engineering depth.

The good news: all three are actively developed, production-ready, and supported by active communities. The barrier to switching is low enough that starting with the pragmatic default (vLLM or TGI) and migrating when you have concrete evidence that a different engine better fits your workload is the right call for most teams.

Monitor your inference engine from day one. The metrics you collect will tell you more than any benchmark, and Stack Pulsar's monitoring guides cover each engine's specific telemetry patterns.