LLMOps: Monitoring AI Features in Production

Your AI feature shipped, the demo was flawless, and everyone moved on. Three weeks later support tickets describe answers that are confidently wrong, latency has crept up, and the token bill has quietly doubled with zero code changes. Nothing paged. That is the defining problem of LLMOps: the system degrades without ever technically failing.

Why traditional monitoring misses LLM failures

Classic observability assumes deterministic software. A request either succeeds or it throws, and your SLOs hang off error rates and latency percentiles. LLM features break that contract in three ways.

• Failures return HTTP 200. A hallucinated refund policy, a retrieval miss, a truncated answer: all of them look like successful responses to your load balancer and your APM.
• The system changes underneath you. Model providers ship silent updates, deprecate snapshots, and adjust safety filters. Your code did not change. Your outputs did.
• Inputs are unbounded. Users will paste contracts, other languages, prompt injections, and 40,000 character rants. There is no schema validation that saves you.

So the question shifts from "is it up?" to "is it still good, safe, and affordable?" Answering that requires four signal families: quality, cost, latency, and safety. Everything below is in service of those four.

Instrument first: trace every LLM call

You cannot evaluate what you did not capture. Before any eval framework or dashboard, get structured traces of every generation. Treat an LLM call like a database query: it gets a span, attributes, and a parent trace.

At minimum, log per call:

• Prompt template ID and version (not just the rendered prompt)
• Model ID and provider, including the exact snapshot
• Input tokens, output tokens, and computed cost
• Latency, split into time to first token and total generation time
• Retrieved context IDs and scores if RAG is involved
• The full response, plus finish reason (stop, length, content filter, tool call)
• A stable user or session ID for feedback joins

The OpenTelemetry GenAI semantic conventions give you a vendor-neutral way to do this, and libraries like OpenLLMetry emit them automatically. A span ends up looking like this:

{
  "name": "chat claude-sonnet-4-5",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.request.model": "claude-sonnet-4-5",
    "gen_ai.usage.input_tokens": 2841,
    "gen_ai.usage.output_tokens": 512,
    "gen_ai.response.finish_reasons": ["stop"],
    "app.prompt_version": "support-answer@v14",
    "app.rag.top_score": 0.71
  }
}

Two hard rules from experience. First, version your prompts like code: a prompt change is a deploy, and every trace must record which version produced the output, or you will never bisect a regression. Second, keep full payloads, but behind access controls and a retention policy, because prompts contain user data and your privacy obligations apply. If your team already runs a tracing stack, extend it rather than bolting on a parallel one; the fundamentals in our guide to logs, metrics, and traces apply directly here.

Evals: the unit tests of LLMOps

Traces tell you what happened. Evals tell you whether it was any good. A production eval program has three layers, and teams that skip a layer always regret it.

Layer 1: offline evals as a CI gate

Build a golden dataset: 50 to 300 real examples with expected properties (not necessarily exact strings). Run it on every prompt change, model swap, or retrieval tweak, in CI, with a hard threshold. Tools like Promptfoo make this a config file:

prompts:
  - file://prompts/support-answer-v14.txt
providers:
  - anthropic:claude-sonnet-4-5
tests:
  - vars:
      question: "Can I get a refund after 30 days?"
    assert:
      - type: llm-rubric
        value: "States the 30 day policy and does not invent exceptions"
      - type: not-contains
        value: "as an AI"
defaultTest:
  assert:
    - type: latency
      threshold: 8000

The gate matters more than the framework. A prompt change that drops your rubric pass rate from 94 percent to 81 percent should fail the pipeline exactly like a broken unit test.

Layer 2: online scoring on sampled traffic

In production you score a sample of live traces asynchronously, typically with an LLM as judge. Judges are imperfect but useful when you constrain them: binary or three-point scales, one criterion per judge, a cheaper model than the one being judged, and periodic human calibration against a few dozen labeled examples. Score dimensions worth automating: groundedness against retrieved context, instruction adherence, format validity, and refusal correctness. Sampling 5 to 20 percent of traffic is a typical starting point; score everything only for low-volume, high-stakes flows.

Layer 3: human signal

Thumbs up or down, edit distance on AI-drafted text the user corrected, escalation to a human agent, task abandonment. These are lagging and sparse, but they are the ground truth that keeps your judges honest. Pipe them onto the same trace IDs so a bad rating links to the exact prompt version, model snapshot, and retrieved chunks that produced it.

Drift: when nothing changed but everything broke

Drift is the silent killer, and it comes from three directions.

• Model drift. Providers update models behind stable aliases. Pin exact snapshots where the API allows it, and when you must migrate, run the golden set against the new snapshot before flipping traffic. Treat a model version bump with the same ceremony as a database migration.
• Data drift. The input distribution moves: a new customer segment, a new language, a marketing campaign that changes what people ask. Monitor input characteristics (length, language, topic clusters via embeddings) and alert on distribution shifts, not just volumes.
• Knowledge drift. For RAG systems, the corpus ages. Retrieval scores decay, chunks go stale, and the model starts answering from parametric memory instead of your documents. Track retrieval hit rate and top-k score distributions over time; a slow slide in median similarity score usually precedes the support tickets by weeks. If your retrieval layer is the weak point, our breakdown of how RAG systems actually work covers the failure modes in depth.

The operational pattern for all three is the same: establish a baseline window, compare a rolling window against it, and alert on statistically meaningful movement in your eval scores or input distributions, not on single bad outputs.

Cost and latency: the observability nobody budgets for

Token spend is a product metric, not a finance line item, because it moves with user behavior and model choice in ways your CFO cannot predict. Instrument it like one.

• Attribute cost per feature, per customer, and per prompt version. "The AI costs a lot" is useless; "the summarize endpoint costs 9x the drafting endpoint because of context stuffing" is actionable.
• Alert on cost per request, not just total spend. Total spend rising with usage is fine. Cost per request rising means a prompt grew, a retry loop appeared, or context windows are bloating.
• Watch output tokens closely. They typically cost several times more than input tokens and they also drive latency, so verbose prompts punish you twice.
• Track time to first token separately from total latency. TTFT is what users feel in streaming UIs; total time is what your timeouts and queue depths feel.

Caching, model routing, and prompt compression can cut spend dramatically, but only if your monitoring can prove which lever worked. We covered those levers in detail in our LLM cost optimization guide; the monitoring described here is what makes that playbook measurable.

Safety and abuse monitoring

Production LLM features get probed. Not always maliciously, but reliably. Your monitoring should treat these as first-class events, not noise:

• Prompt injection attempts, especially in RAG and agent systems where retrieved content or tool outputs can carry instructions. Log and classify them; a spike is reconnaissance.
• Content filter triggers and refusals. A rising refusal rate can mean an attack, a provider-side filter change, or a legitimate new use case you are blocking. All three deserve investigation.
• PII in prompts and outputs. Run lightweight detection on traces before they land in long-term storage, and redact at ingestion.
• Agent tool calls. If the model can call tools, every call needs an audit log with arguments, and destructive tools need allowlists and human approval thresholds. This is where LLMOps meets security engineering, and blast radius thinking applies directly.

Choosing the tooling

The landscape is crowded but the shapes are stable. Langfuse (open source, self-hostable) and LangSmith are trace-first platforms with eval and prompt management built in. Arize Phoenix is strong on drift and embedding-space analysis. Helicone sits at the proxy layer and gives you cost and caching with minimal code. Datadog, Grafana, and New Relic now ingest GenAI traces if you want LLM signals inside your existing observability stack.

Decision framework, in order:

• If you have strict data residency needs, self-host: Langfuse or Phoenix.
• If your org already lives in Datadog or Grafana, extend it and add a lightweight eval runner like Promptfoo in CI.
• If you are a small team shipping fast, a managed platform buys you dashboards and judges on day one.
• Whatever you pick, emit OpenTelemetry GenAI conventions so the choice stays reversible.

The tool matters less than the discipline: versioned prompts, a golden set, a CI gate, and sampled online scoring. Teams with that discipline and a spreadsheet outperform teams with a fancy platform and no baseline.

A production readiness checklist

Before an LLM feature earns real traffic, walk this list:

1. Every LLM call emits a trace with model snapshot, prompt version, token counts, cost, and finish reason.
2. A golden dataset of at least 50 real examples exists and runs in CI with a hard pass threshold.
3. Prompt changes go through pull requests and are versioned; no editing prompts in a dashboard on Friday afternoon.
4. Online judges score a defined sample of production traffic on groundedness and instruction adherence.
5. User feedback (explicit or behavioral) is joined to trace IDs.
6. Alerts exist for: eval score drop, refusal rate spike, cost per request increase, TTFT regression, and retrieval score decay.
7. Model snapshots are pinned, and a documented migration procedure exists for version bumps.
8. Payload logging has retention limits, access controls, and PII redaction.
9. A rollback path exists for prompts and models, independent of an app deploy.
10. Someone owns the weekly review of failed evals and negative feedback. Unowned dashboards decay into wallpaper.

If you cannot check items 1, 2, and 6, you do not have an AI feature in production. You have an AI feature in the wild.

How Innovation T can help

Innovation T designs, builds, and operates LLM-powered products for clients across Europe, the Gulf, and North Africa: RAG systems, AI agents, and the observability stack that keeps them honest. We set up the tracing, eval pipelines, drift alerts, and cost guardrails described here as part of delivery, not as an afterthought, so your team inherits a system it can actually run.

If you are shipping an AI feature, or you already shipped one and the silence is making you nervous, explore our software and cloud engineering services or talk to our engineers. We will tell you within one call whether your monitoring would catch the failures that matter.