- AgentOps
AgentOps: the 12 metrics to monitor once an Agent is in production
By Apolla delivery team2 min read
Key takeaway
Once an Agent is in production, monitor at least 12 metrics across four categories: usage (call volume, active users, department distribution), effectiveness (accuracy, task completion rate, resolution rate, satisfaction), cost (token consumption, cost per session, model distribution) and risk (sensitive content hits, unauthorized access attempts and anomalous output). 88% of production Agent deployments have had an incident, and more than 40% of Agentic projects are expected to be cancelled before 2027. In most cases the cause is missing governance and unclear ROI, not the technology.
Why go-live is only the beginning
Models get updated, knowledge goes stale, and the way users phrase questions changes. An Agent that launches at 90% accuracy and is left alone for three months commonly drops to 75%. Without monitoring you cannot see it degrading; without evaluation you cannot tell whether a change made it better or worse.
- 88%
- of production Agent deployments have had an incident
- >40%
- of Agentic projects expected to be cancelled before 2027
- 12metrics
- the core set we recommend monitoring
Gartner; industry Agentic AI 2026 survey
Usage: is anyone using it?
- Call volume: conversations per day and per hour, to spot weekday patterns and abnormal peaks and troughs
- Active users: weekly and monthly actives, plus new and churned users
- Department distribution: which departments use it and which bought it but do not; the latter is where training and operations should start
Effectiveness: is it working well?
- Answer accuracy: sampled human labeling or evaluation set scores, weekly
- Task completion rate: of tasks that require tool calls, the share executed successfully end to end
- Resolution rate: whether the user's problem was actually solved, judged from end-of-session feedback or subsequent behavior
- Satisfaction: thumbs up and down, plus voluntary ratings
Why resolution rate matters more than accuracy
An "accurate" answer is not a "solved" problem. A customer service Agent that accurately tells the user "please contact a human" has 100% accuracy and 0% resolution. Answering correctly is only the first step; resolution rate is the final test of value.
Cost: how much is it spending?
- Token consumption: broken down by model, by application and by department
- Cost per session: identify unusually expensive session patterns (such as repeated retries with long context)
- Model distribution: the ratio of calls to cheaper versus stronger models, and whether the routing strategy is working
Risk: has anything gone wrong?
- Sensitive content hits: how often input and output filters trigger, and on what
- Unauthorized access attempts: the number of requests for data the user has no permission to see
- Anomalous output: format errors, hallucination flags, failed tool calls
The monthly operations cadence
Review the monitoring dashboard, run evaluation scores, attribute root causes for high-frequency issues, optimize prompts and workflows, review costs, ship the next version, and publish a monthly operations report. For the first 3–6 months the delivery team leads and transfers knowledge; after that it shifts to advisory support.