Blog
Deep dives on LLMOps, FinOps, Kubernetes, and AI infrastructure.
Mesh Inference on iroh: GPUs in Three Offices and a Closet
Mesh LLM turns scattered GPUs into one OpenAI-compatible API. Skippy split-mode layer-pipeline inference behind localhost:9337/v1 — and the OTel gap.
OpenAI Just Made Your Agent a Black Box (and What to Do)
Codex multi-agent-v2 encrypts the parent→subagent payload. The agent-side OTel proxy that keeps your plaintext copy before Responses encrypts it
Custom AI Silicon 2026: Meta MTIA, Trainium2, TPU, Maia
Meta MTIA 300/450/Iris, AWS Trainium2/Inferentia2, Google TPU v5e/v6, Microsoft Maia 100 vs NVIDIA H100/B100 — vendor-neutral, dollar-per-token.
Per-Engineer AI Observability 2026: Beat Reflection
OTel + LangSmith + ClickHouse reference schema for per-engineer Claude observability — session cost, four-signal dashboard, 1/3/6/12-month retention.
Get the best of this blog, weekly
The Stack Pulse: LLMOps, FinOps, and AI infrastructure intelligence. No fluff, no vendor pitches.
Agent Observability at 1,200+ Agents: OTel Stack 2026
AWS AgentCore vs. OSS for 1,200-agent fleets: OpenTelemetry gen_ai.* conventions and a Tempo / ClickHouse / OpenSearch trace-store comparison.
Agent Sandbox vs. Agent Substrate: CNCF Runtime 2026
Two CNCF SIG Apps projects — agent-sandbox and agent-substrate — and the OTel pattern that makes both debuggable in production.
Coding Agent Cost Observability 2026: One View
How a normalized OTel GenAI schema caught $1,600 of hidden Claude Code + Cursor overlap on an $11,400 bill — five-panel Grafana cost view.
MCP Enterprise Authorization 2026: The Missing Auth Layer
Cloudflare, Auth0, Stytch and Stargate ship MCP server auth primitives in 2026. OBO flow, OAuth scopes, audit logs, rate limits, residency.
The Agentic Harness for AI Incident Response
PagerDuty CAIO: AI incident tools are missing a critical layer. The 4-layer harness (state, memory, authority, verification) for end-to-end agent ownership.
Multi-Dimensional AI Retrieval: Beyond Vector Search
Vector search is 30% of production RAG. The other 70% is BM25 + cross-encoder rerank + tensors + rules. The 2026 stack, plus 3 anti-patterns to avoid.
AWS FinOps Agent 2026: The First Frontier Agent for FinOps
AWS FinOps Agent investigates cost anomalies in plain English inside Slack and Jira. The frontier-agent pattern, what it covers, and the FinOps maturity model.
Carbon-Aware AI Inference 2026: Cut Energy 30-50%
A 1000-H100 inference fleet spends $850K/year on electricity. Continuous batching, KEDA queue-depth autoscaling, and Kube-Green recover 30-50% of that energy.
Beyond the Stack Trace: AI Debugging Paradigm
Why stack traces fail for non-deterministic AI. The 4 prompt-trace primitives, 5-step workflow, and verifier pattern that replaces the stack trace in 2026.
AI Operational Debt 2026: 3 Forms That Break AI Strategy
Prompt debt, eval debt, tool debt: the three forms of operational debt unique to AI systems, and the audit pattern that finds them before they break production.
AI Cost by Workflow 2026: The Tokenmaxxing Layer
Per-workflow token attribution: tag every LLM call with workflow_id, build per-business-process cost dashboards, route workflows to cheaper models.
Agentic Ops Platform 2026: Enterprise Reference Architecture
Enterprise architecture for 200+ internal AI agents: per-agent RBAC, audit logs, sandboxed tools, prompt-injection defense, and the Kubernetes operator pattern.
Probabilistic Observability 2026: AI Debugging Discipline
The 4 primitives for debugging non-deterministic AI: output distributions, semantic traces, statistical regression, hallucination-as-metric. OTel + Grafana.
Inference API Gateways 2026: LiteLLM vs BentoML vs Ray Serve
A practitioner's comparison of three inference gateway and serving stacks — LiteLLM, BentoML, and Ray Serve — when to use each, and the limits.
AI Coding Agent FinOps 2026: Copilot, Cursor, Devin Cost
Per-engineer token costs, per-LOC and per-PR attribution, anomaly detection, and enterprise policy for AI coding agents: Copilot, Cursor, Devin.
The Google Remy Leak: AI Agent Stack Risk in 2026
Google's Gemini Workspace agent stack leaked via OAuth over-scoping, calendar side-channels, and draft-state recovery — a pattern, not a single CVE.
eBPF for AI Networking: GPU Workload Visibility
How eBPF, Cilium, and Hubble deliver kernel-level observability for AI infrastructure: GPU scheduler events, NCCL/RDMA latency, and inference pod traffic.
Backup and Restore for Vector Databases: A Production Guide
How Pinecone, Weaviate, Qdrant, Milvus, and Chroma handle backup, restore, PITR, and disaster recovery — with concrete RTO/RPO numbers and a runbook.
Trainium2 vs Inferentia2: When AWS Custom Silicon Beats H100
2TB HBM fits Llama 70B on one Trainium2 host: EKS + SageMaker numbers, NeuronLink collectives, Neuron SDK compile times, $0.30/M-token Inferentia2 vs H100.
Arize Phoenix 15.4.0: Open Source LLM Observability
A practitioner's guide to Arize Phoenix 15.4.0: embedding drift detection, RAG trace analysis, the agent toolset, and wiring it into a self-hosted LLM stack.
vLLM vs SGLang vs Ollama 2026: Production Comparison
vLLM tops throughput, SGLang adds multi-model routing, Ollama wins on simplicity. Benchmark data and decision framework for 2026 self-hosted LLM serving.
AI SLO/SLA Contracts: A Practical Guide for Infra Teams
TTFT p99 targets, composite SLA math, model deprecation clauses, and RAG recall SLOs — a practitioner's guide to AI service SLAs in production.
Multi-LLM Routing: Cut Costs 40% Without Quality Loss
200K requests across GPT-4o, Claude 3.5, and Gemini 2.0 Flash: cost-plus-latency routing saved 44% with no measurable quality drop. Architecture inside.
Fine-tuning in Production: The Infrastructure Guide for 2026
Axolotl, Unsloth, TRL, QLoRA and the eval pipeline that catches bad checkpoints — the fine-tuning stack that actually works in production in 2026.
AI Agent Reliability 2026: Failure Modes + Observability
What actually breaks an agent at 2am: four failure modes, the OTel span hierarchy that surfaces them, and the CrewAI v0.5 + LangGraph fixes that work.
SGLang Production Monitoring: A Complete Practical Guide
Monitor SGLang in production: RadixAttention architecture, KV cache metrics, prefill/decode throughput, TTFT, Prometheus + Grafana instrumentation, and a frank comparison with vLLM and Ollama.
Build Your First LLM Monitoring Stack: OTel + Prometheus
A practical guide to instrumenting LLM applications with OpenTelemetry, scraping metrics with Prometheus, and visualizing token costs, latency, and quality signals in Grafana dashboards.
Multi-Modal LLM Monitoring in Production: A Practical Guide
How to monitor vision, audio, and text inputs in multi-modal AI systems. Covers metrics unique to multi-modality, OpenTelemetry instrumentation patterns, and the monitoring stack for production MLLM applications.
LLM Monitoring Dashboard Templates: Grafana + Prometheus
Production-ready Grafana dashboard JSON and Prometheus queries for LLM monitoring. Token throughput, TTFT/TPOT latency, cost attribution, error rates, and context window utilization — all in one template.
LLM Context Window Optimization: Cut Costs, Keep Quality
A practical guide to reducing LLM inference costs by 40-70% using semantic truncation, context compression, dynamic sizing, and hybrid retrieval — with code examples.
AI Model Monitoring vs Traditional APM in 2026
Four fundamental differences between AI and software monitoring — non-deterministic output, token-based cost, multi-component latency, and stateful context windows.
LLM Evaluation Frameworks: RAGAS, TruLens, and the Stack
RAGAS scores retrieval faithfulness, TruLens traces attribution per output. Comparison table, CI gating pattern, and the threshold strategy we use.
LLMOps Platform Comparison 2026: Guide to the Leading Tools
Hands-on 2026 LLMOps comparison: Braintrust's eval gating vs LangSmith tracing vs Phoenix OTel — pricing tiers, PII gaps, and which scales further.
Prompt Injection: Detection and Prevention Strategies
Prompt injection is an active threat in production AI systems. Here are the detection methods, prevention strategies, and the defense-in-depth architecture you need to stay protected.
vLLM vs TGI vs TensorRT-LLM on H100s: The Benchmarks
vLLM, TGI, and TRT-LLM benchmarks plus Qwopus3.6 and Neutrino-8B serving paths, VRAM, quantization, and engine support.
Monitor LLMs Without Per-Token Fees: 5 Open-Source Tools
Replaced our $4,200 Helicone bill with OTel + Prometheus + Grafana + Loki + Tempo — $180 flat at 10B tokens/month, PagerDuty SLOs, vLLM p99 1.4s TTFT.
LLM Latency Monitoring 2026: TTFT and TPOT
Every millisecond your users wait for an LLM response, engagement drops. Here is how to measure, monitor, and fix LLM latency with TTFT, TPOT, and the metrics that actually matter in production.
Prometheus vs Grafana: Fix Alert Fatigue and Unknown Pods
Your Kubernetes pods are eating through budget, your on-call is drowning in alert fatigue, and your dashboards show 'unknown' for half your services. Here is exactly how to fix your observability stack using Prometheus and Grafana — and when to use each one alone.
LLM Hallucinations: Five Production Detection Methods
Five detection methods with thresholds: regex PII gates, RAGAS faithfulness >0.80, sentence-transformer drift <0.35, judge model, Prometheus SLO burn alerts.
LLMOps Observability: Latency, Hallucinations, and Drift
A blueprint for LLMOps observability: why HTTP 200 is a lie for LLM apps, the three pillars of LLM health (latency, quality, reliability), and how to implement an LLM Health Score for production AI systems.
vLLM vs Triton: Real H100 Throughput and Migration Cost
PagedAttention gives vLLM 2-5x throughput vs Triton on Llama-3 70B H100; Triton wins for multi-model serving. FP8, AWQ, ops, and TTFT compared.
AI Incident Postmortem Template: Four-Question Framework
When your AI system fails, you need answers fast. This copy-paste postmortem template uses a proven four-question framework — with a real medical AI incident example and a production runbook checklist your team can use immediately.
Kubernetes GPU Operator: A Production Setup Guide
GPU pods not scheduling? The NVIDIA GPU Operator stack plus HAMi (CNCF Incubating Jul 2026): sharing primitives, migration cookbook, cross-vendor scheduler.
How to Monitor Ollama in Production: The Observability Stack
A 4-hour outage from a silent Ollama CPU fallback: TTFT 30x slowdown, every probe green. The Prometheus scrape config, VRAM alerts at 90%, and 5 Grafana panels.
DevOps Supply Chain Security 2026: CPU-Z Compromise Lessons
The April 2026 CPU-Z/HWMonitor supply chain attack exposed how even trusted developer tools can become attack vectors. Here's what infrastructure and DevOps teams need to know about software provenance, SBOM, and supply chain hardening.
OpenTelemetry for AI Inference: Tracing LLM Pipelines
How to instrument LLM inference pipelines with OpenTelemetry — from prompt injection to token streaming, from model serving to downstream tool calls, using OTel's AI semantic conventions.
LiteLLM Production Monitoring 2026: Gateway + Cost Tracking
Monitor LiteLLM 1.95.0-dev.1: critical SSRF fix, Rust gateway routes, MCP DCR, spend controls, signed images, and the current stable pin.
LLM Model Drift Detection 2026: Monitoring AI Degradation
Our support LLM silently degraded for weeks before any alert fired. The 4-layer drift stack I built: statistical, embedding, LLM-as-judge, reference tests.
LLM Incident Postmortem 2026: Lessons from AI Failures
Real incident retrospectives from legal RAG, medical AI, and customer support AI failures. Learn the four-question AI postmortem framework, the failure modes unique to non-deterministic systems, and the runbook patterns that prevent repeat incidents.
SRE Best Practices for AI/LLM Systems in 2026
A practical SRE playbook for operating AI and LLM systems in production. Covers AI-specific SLOs, SLIs, error budgets, incident response runbooks, on-call procedures, and chaos engineering for AI workloads.
Terraform vs Pulumi: AI Infrastructure Decisions
Terraform vs Pulumi for AI/ML infra: dynamic GPU clusters, Kubernetes, multi-cloud routing, and the Terraform 1.17 alpha caveat.
LLM Security Hardening 2026: A Defense-in-Depth Guide
Prompt injection, jailbreaking, and model extraction threaten production AI systems. Here are the six defense layers every AI engineer needs in their production stack.
Helicone vs Portkey vs LangSmith: LLM Observability 2026
Helicone (100K/mo free), Portkey ($50/mo at 500K routed), LangSmith Plus ($39/seat). Proxy tracing, eval, and guardrails tested on real production LLM traffic.
The Rise of eBPF 2026: A New Era for System Observability
eBPF is rewriting the rules of Linux observability. Learn how extended Berkeley Packet Filter programs enable kernel-level monitoring without instrumentation, and why it matters for AI infrastructure.
Datadog Alternatives 2026: 5 Cost-Effective Picks
Five Datadog alternatives that deliver better value at scale — covering LLM inference, Kubernetes, and cloud cost observability for 2026.
Monitoring LLM Hallucinations 2026: AI Engineer Guide
Hallucinations are the blind spot of LLM monitoring. Here is how to detect, measure, and reduce them in production — with rule-based checks, LLM-as-a-judge, and embedding drift detection.
K8s GPU Scheduling: Stop NUMA Crossings Killing Training
NUMA crossings killed 38% of our 70B training throughput (11d → 7d with NVLink pinning). The K8s GPU scheduling playbook for real perf.
OpenClaw Reliability: Production AI Agent Patterns
A senior SRE perspective on OpenClaw failure modes in AI agent production environments, with hardening patterns and monitoring strategies for DevOps and AI engineers.
Multimodal LLM Cost Optimization 2026: Vision and Audio AI
GPT-4V costs 4x GPT-4o, a 1024px image can burn 16K tokens. What cut our bill 62%: image compression, region cropping, Whisper-tiny, prompt caching.
AWS Savings Plans vs Reserved Instances: 2026 FinOps Guide
AWS Compute Savings Plans beat EC2 RIs for GPU inference. Coverage math, Auto-Refit pattern, and the ml.p5 commitment trap I learned the hard way.
Cut vLLM GPU Costs 40% with KEDA Queue-Depth Autoscaling
I cut our vLLM GPU bill 40% (8xH100, $11.4k → $6.8k/mo) by replacing HPA with KEDA queue-depth autoscaling. Prometheus + Karpenter config inside.
Datadog Migration: From $15K to $3K/mo Playbook
Three client migrations from Datadog to Grafana + Prometheus + Tempo. Average savings: 80%. This playbook covers the exact sequence (don't skip order), the billing traps that inflate your new stack, and the three dashboards you need first.
State of AI Infrastructure 2026: From Hype to Production
A practical analysis of the AI infrastructure landscape in 2026 — GPU providers, inference frameworks, SLM adoption, and the FinOps reality check that follows scale.
Agentic Observability: Multi-Agent LLM Monitoring
A practical guide to observability for agentic AI systems — step-level tracing, cost accounting, reliability monitoring, and the four-layer stack you need to debug production agents.
GPU Monitoring for AI Inference: A Practical Guide for 2026
Monitor GPU utilization, VRAM, temperature, and power draw for AI inference. Covers DCGM, Prometheus, Kubernetes GPU scheduling, MIG partitioning, and cost optimization.
RAG Observability 2026: What Matters in Production
A practical guide to monitoring RAG pipelines in production — retrieval precision, context utilization, answer faithfulness, embedding drift, and the metrics that actually predict user satisfaction.
Agentic AI Infrastructure for DevOps and Platform Engineers
From stateless LLM calls to autonomous multi-step agents — a practical guide to the infrastructure patterns that make agentic AI production-ready.
Kubernetes Cost Optimization: Cutting Cloud Bills in Half
Practical strategies to cut Kubernetes spend by 40-60%: right-sizing nodes, Spot instance mixing, cluster autoscaling, namespace quotas, storage tiering, and Kubecost for visibility.
MCP Monitoring: Observability for Model Context Protocol
A practical guide to monitoring MCP (Model Context Protocol) servers in production. Covering metrics, dashboards, alerting rules, and open-source tooling for 2026.
LLM Observability: Complete Implementation Guide
A practical guide to implementing LLM observability in production. Covers the 8 critical signals, OpenTelemetry instrumentation architecture, and the monitoring stack your AI applications need at scale.
Catch GPU Throttling at 83°C: Prometheus + Grafana + eBPF
Above 83°C, vLLM pods throttle silently until OOMKilled. Prometheus + Grafana + eBPF catches GPU starvation across 200 GPU pods before it cascades.
Vector Database Comparison 2026: Pinecone, Milvus, Weaviate
Pinecone vs Milvus vs Weaviate for production RAG: performance, cost, hybrid search, backups, and Weaviate 1.38.8 fixes.
The State of Observability in 2026: Trends and Tech
Why p99 2.3s no longer fires alone in 2026: semantic observability ties alerts to user impact across eBPF, OTel, and vLLM KV cache at 95%.
Cloud FinOps in 2026: From Chaos to Controlled Spend
A practical guide to cloud waste reduction without sacrificing performance — covering tagging strategies, reserved capacity, and cost-aware architecture.
LLM FinOps 2026: Cut Your AI Bill, Keep Performance
A practical guide to reducing LLM inference costs by 60-80% using tiered model routing, semantic caching, prompt optimization, and self-hosting — without measurable accuracy loss.
vLLM Production Monitoring: A Practical Stack Guide
GPU cache utilization, KV cache hit rate, TTFT/TPOT metrics, and a complete Prometheus + Grafana monitoring setup for vLLM inference servers — updated for v0.19.
Monitoring the Unseen: Observability for AI/ML Pipelines
LLMs, vector databases, and RAG pipelines introduce new failure modes. Here is how to instrument your AI stack for production reliability.
LLM Cost Monitoring Tools 2026 — Complete Guide
Stop guessing where your LLM spend goes. This guide covers the full-stack approach to monitoring LLM costs — from token-level attribution per user and model to real-time alerting on budget overruns and anomaly detection.