From MLOps to LLMOps: Operating AI Systems in Production
Non-deterministic systems need evaluation as a release gate. Without it, every model update is an unmonitored change.
Teams that have run machine learning in production have most of the operational instincts required for LLM systems, and one assumption that no longer holds: that the same input produces the same output, and that the model changes only when you change it.
Neither is true with hosted language models. That single difference reshapes testing, release, and incident handling.
Evaluation is the release gate
Without an evaluation suite there is no way to know whether a change improved or degraded the system, so every deployment becomes a guess validated by anecdote.
Build a set of representative cases — typical requests, known hard cases, and previously reported failures — and score each run on the dimensions that matter: factual accuracy against the source material, adherence to required format, policy compliance, and task success. Automate scoring where possible with deterministic checks and model-based grading, and keep a smaller human-reviewed subset for judgement-heavy criteria.
Then run it on every change: prompt edits, retrieval configuration, model version, and tool definitions.
Version everything that affects behaviour
| Artefact | Why it must be versioned |
|---|---|
| Prompts and system instructions | Primary determinant of behaviour; edits are code changes |
| Model name and version | Behaviour differs materially between versions |
| Retrieval configuration and index version | Changes what the model sees |
| Tool definitions and schemas | Changes what the system can do |
| Generation parameters | Affects variability and determinism |
Prompts edited in a vendor console with no version history are the most common operational weakness in production AI systems. Treat them as source code, reviewed and deployed through the same pipeline as everything else.
Observability for probabilistic systems
Log each interaction with inputs, retrieved context, model and prompt version, output, latency, cost, and any downstream user action — accepted, edited, or rejected. That user action signal is the most valuable quality metric you will have, because it reflects real usefulness rather than benchmark performance.
Alert on shifts in refusal rate, output length distribution, latency, cost per request, and retrieval hit rate. Sudden movement in any of these usually indicates an upstream model change, a broken index, or an abusive traffic pattern.
Handle model updates as external change events
Pin versions where the provider permits it, and maintain a process for adopting new ones: run the evaluation suite, compare against baseline, canary a small traffic share, then roll forward. Where pinning is unavailable, increase production monitoring sensitivity and accept that you are consuming a moving dependency — which should influence how critical a process you place on it.
Keep a rollback path that includes prompt and configuration state, not only application code.
Incident handling looks different
AI incidents are frequently quality incidents rather than outages: the system is available and producing wrong or inappropriate output. Define severity in terms of impact — incorrect customer-facing information, policy breach, data exposure — and prepare responses beyond a code fix: disabling a capability, reverting a prompt version, tightening a guardrail, or falling back to a deterministic path.
Build that fallback deliberately. A system with no non-AI degraded mode cannot support a critical process.
Close the loop with feedback
Capture user corrections and rejections, review a sample weekly, and feed genuine failures back into the evaluation suite as new cases. Systems that do this improve steadily; systems that do not degrade quietly as usage patterns drift away from the cases the prompts were originally written for.
Frequently asked questions
What is LLMOps?
The practice of deploying, evaluating, monitoring, and maintaining LLM-based systems — versioning prompts and configuration, automated evaluation, output observability, cost control, and managing provider model changes.
How do we test non-deterministic systems?
With an evaluation suite of representative cases scored on accuracy, format, policy, and task success, run on every change against a recorded baseline.
What happens when the provider updates the model?
Behaviour can change without any change on your side — pin versions, evaluate new ones before adopting, and monitor production quality continuously.
What is the most overlooked practice?
Version-controlling prompts. Editing them in a console with no history removes any ability to explain or reverse a quality regression.