Count the decisions baked into a dashboard.
Someone chose which metrics matter. Chose how far to roll the numbers up. Chose the percentile. Chose which dimensions to split out and which to fold together. Chose the time range. Chose what goes at the top. Chose which fifteen of your four hundred services get a panel.
Every one of those choices threw information away. That was the point. A dashboard exists because a human can take in maybe a dozen visual signals at once. So you squeeze millions of events into twelve pictures and hope you squeezed along the right lines.
Same story with alerts. A threshold exists because nobody can watch a metric continuously, so you pick a number that means “wake someone” and accept that anything below it goes unnoticed. The threshold says nothing about the system. It says that human attention is expensive and comes and goes.
Both are excellent engineering. Both are answers to limits that the main reader of your telemetry no longer has.
A dashboard is a lossy compression of telemetry, tuned for the bandwidth of human vision. When the operator is an agent, you’re handing it the thumbnail and throwing away the photograph.
This is part nine of Rethinking the SDLC. The previous piece put agents into the incident loop with explicit autonomy tiers. This one is about what they need to see — and why “give the agent access to Datadog” is not the answer.
What agents don’t need, and what they desperately do#
Let’s clear away the part everyone gets wrong first.
Agents don’t need a picture that sums things up. They don’t need a percentile chosen for them. They don’t need the top-fifteen-services rule of thumb. They don’t need pretty. They can hold more raw dimensions in memory than any human, match patterns across services without eye strain, and read every log line instead of the few that got past a sampling rule.
But they have a different and serious limit that almost nobody designs for: they cannot work out the things your engineers know unless someone tells them.
A senior engineer looking at a latency spike in the notification service knows it’s downstream of the queue, knows the queue was migrated last month, knows the team that owns it, knows the deploy went out forty minutes ago, knows this pattern preceded the outage in March, and knows finance notices this one before engineering does. None of that is in the telemetry. It is in their head, and they use it in about two seconds without noticing.
Give an agent the same latency spike with no topology — no map of which service calls which — no deploy history, no ownership, no record of past incidents, and it will produce a confident, believable root cause that nobody can disprove. It will name the service the metric came from, because that is the only thing it was given.
That is the real failure mode of agentic observability, and it does not look like a failure. It looks like a fast answer. Wrong root causes at machine speed are worse than no answer, because someone acts on them.
It always takes the same shape in practice, and it is worth seeing in the concrete. A queue backs up. The agent has the metrics, so it correctly identifies which consumer slowed and when. Then it produces a root cause naming that consumer’s own code — because the consumer is the only entity in its context. What actually happened was a config change to a shared connection pool forty minutes earlier, in a different service, owned by a different team. The agent never stood a chance. The config change was not an event on the same timeline as the metrics, the pool was not in any dependency graph it could read, and the ownership line existed only in the org chart.
Notice what was missing there. Not data — the metrics were complete, with nothing sampled away. What was missing was one link: from a change event, to the resource it changed, to the services that depend on it. Every one of those three facts existed somewhere in the company, in three different systems, and a human on-call would have joined them from memory in about four seconds.
So the work is not giving agents more data. Volume is the easy part. The work is writing the relationships down.
The operational context contract#
Here is what telemetry has to carry when a machine reads it first. Seven things. Most systems have two or three.
1. Events, unsampled, structured, and queryable.
Sampling was a way to save money, built on the assumption that a human would read the result. Nobody was going to read a million log lines, so why store them. Agents will read a million log lines. And structure matters more than volume: a structured event with typed fields is a fact, while a log line is a string that has to be parsed and guessed at again on every read.
The key requirement is not storage, it is query. An agent should be able to ask exact questions and get exact answers, not stand in front of a firehose. This matters more than it sounds: Anthropic’s engineering work on tool use notes that tool definitions alone can consume ~55k tokens before any work begins. Context is a fixed budget. An agent that has to swallow a gigabyte to answer one question will run out of room before it reaches an answer. Dumping telemetry on an agent is the same mistake as dumping it on a human, one level up.
2. Topology — who calls whom, right now.
Service dependencies, as data, kept current — not a Confluence diagram from 2024. With a direction. And with the kind of dependency named: direct call, queue, shared database, event subscription.
This is the single most valuable thing most organizations can add, because it turns “these two things moved together” into “this one probably caused that one.” Without topology, an agent can only tell you two metrics moved together. With it, the agent can tell you which one likely caused the other, and which way the damage spread.
3. Causality — traces, and change events as first-class citizens.
Distributed traces are the obvious half. The half everyone forgets is that every change is an event, and it belongs on the same timeline as your metrics: deploys, config changes, flag flips, migrations, scaling actions, infrastructure applies, and — most of all — agent actions.
Most organizations have deploy history in one system, flag changes in another, infrastructure applies in a third, and agent activity in a fourth or nowhere. A human bridges those systems by remembering. An agent cannot, and should not be asked to. Put every change on one timeline with the telemetry, and finding the root cause stops being detective work and becomes a lookup.
4. Ownership, as structured metadata.
Every service, every table, every queue, every job: who owns it, who is on call for it, how critical it is, who to notify, and what it costs the business when it fails. This has always been recommended and always been optional. It is load-bearing now. It is the difference between an agent reaching the right human in thirty seconds and an agent posting into a channel nobody reads.
5. Declared expected behavior.
SLOs, error budgets, latency budgets, throughput baselines, expected traffic shapes by hour and day. Written down, machine-readable, versioned with the service.
This is what lets an agent ask “is this abnormal?” instead of “is this above the number someone picked in 2023?” And notice where these come from. They are the same invariants and constraints that belong in your intent packages and your verification requirements. Written-down expected behavior is the artifact that ties what you intended at build time to what actually happens at runtime — which is the whole loop this series has been building toward.
6. Precedent — incidents and remediations as queryable history.
“Have we seen this before, and what fixed it?” is the most valuable question in incident response, and the hardest to answer. The answer lives in postmortem documents written for humans, and in the memory of whoever was on call that night.
Structured incident history — symptoms, services hit, the real cause, the fix applied, whether it worked — turns company memory into a lookup. This one builds up over time: every incident you record properly makes the next one faster, forever, and it survives people leaving.
7. Machine-readable runbooks.
Not prose. Structure: what normal looks like, the known ways it fails and how to tell them apart, the diagnostic steps with real commands or tool calls, the fix for each failure mode, the autonomy tier for each fix, and what must never happen without waking a named person.
That last field is what makes a runbook usable rather than decorative
And a warning that follows straight from that: an agent reads a runbook as fact. Humans read documentation with suspicion. They squint at a page that contradicts the code and go ask someone. That doubt was an error-correction layer nobody ever named, and agents don’t have it. A stale runbook is no longer a small annoyance. It is wrong instructions carried out with full confidence at 3 AM. This is why knowledge infrastructure has to be owned and current rather than left to pile up, and why Stripe’s agents pull context from a central Toolshed MCP server exposing 400+ internal tools instead of scraping a wiki. The quality of your context sets the ceiling on the quality of your agents. Build it like a platform.
What happens to dashboards and alerts#
Not dead. Demoted, and given a clearer job to do.
Dashboards become a place for humans to judge, not a way to find problems.
When an agent brings you a situation, you need to see it — quickly, in a form that supports a decision. That’s a dashboard, and it’s a good use of one. What dashboards should stop being is the way problems get found. Nobody should be discovering incidents by staring at a wall. The wall is for understanding what you have already been told.
In practice that means far fewer dashboards, built for a situation instead of maintained forever. Most organizations carry hundreds of stale dashboards nobody reads, each one a snapshot of what somebody once worried about. Delete them without ceremony.
Alert thresholds become routing rules, not detection rules.
This is the sharper change. If an agent watches every signal all the time, finding problems no longer needs a threshold. Thresholds existed to protect human attention. What still needs a decision is routing: which findings are worth waking a human for, which get handled on their own, which go into a morning report, which just get logged.
That is a different design problem, and it fails in a different way. Tuning thresholds was about not missing things. Designing routing is about not flooding the humans with things the machines should have handled. And autonomy tiers govern it, not numbers. The question is “what may be handled without a person?”, which you have already answered per workflow.
The cost case for high cardinality flips.
High-cardinality dimensions — the ones with thousands of distinct values — were expensive partly because they blew up storage and partly because they were useless on a chart. You cannot draw ten thousand lines. The second reason is gone. An agent does not care that a per-customer, per-endpoint, per-version breakdown cannot be drawn. That is exactly where the answer usually is.
So the sums change. Some of the detail you were throwing away because it was unusable is now your most valuable data. That does not mean keep everything. It means the reason you were throwing it away needs a second look, and “we couldn’t chart it” is no longer a reason.
The part that will annoy people: this is a data-modeling project#
Here’s the uncomfortable conclusion, and it’s why I think most organizations will do this badly.
You cannot buy any of the above. No product will make your topology accurate, your ownership metadata complete, your change events land on one timeline, or your runbooks structured and true. Vendors will sell you an AI layer on top of the telemetry you already have, and it will produce smooth, confident analysis of a context that is missing almost everything it needs — the worst of both worlds, because the smooth writing hides the gaps.
Observability for machines is a data-modeling and ownership problem in a tooling costume. The work is:
- Deciding what things exist in your system and giving each a stable identity.
- Recording relationships between them, and keeping the record current automatically.
- Putting every state change on one timeline.
- Attaching ownership and consequence to every entity.
- Declaring expected behavior next to the code that implements it.
- Structuring what you know so it can be queried instead of read.
That is a platform investment with nothing to demo. It is also the difference between agents that diagnose correctly and agents that produce fluent nonsense at speed — and you will not know which kind you have until an incident tells you.
Six moves#
- Put deploys, flags, config, migrations, and agent actions on one timeline with your metrics. If you do one thing from this article, do this. Most root-cause work is joining “what changed” to “what got worse,” and in most organizations that join is a human with two browser tabs.
- Make topology queryable and keep it current automatically. Build it from traces or service registration, not by hand. A hand-maintained dependency graph is wrong within a month, and worse than having none, because agents will believe it.
- Complete ownership metadata for your top fifty services. Owner, on-call route, how critical it is, what it costs the business. Boring, unglamorous, and it is what makes every escalation land on the right person instead of roughly the right team.
- Convert your five most-used runbooks to structured form. Failure modes, diagnostics, fixes, an autonomy tier per fix, hard stops. Then feed them to an agent as context and watch what it gets wrong. That is your documentation audit, and it is more honest than any review.
- Expose telemetry as tools, not as a firehose. Exact queries an agent can call, returning small structured answers. Being able to ask beats having a lot of data every time, because context is the scarce resource.
- Delete your unused dashboards. Check the view counts. The ones nobody has opened in six months cost you upkeep and mislead the next person who digs. Removing them is a real improvement.
We built a window when we needed a model#
The thing I keep noticing about this change is how often the answer is “the thing we always said we should do, and never quite got round to.”
Accurate service topology. Complete ownership metadata. Unified change timelines. Structured runbooks. Declared SLOs. Every one of those has been on best-practice lists for a decade, and every one lost the priority argument to something more visible, because a human could make up for it by knowing things.
That layer of human compensation is what is being removed. Not because humans are leaving, but because humans are no longer the first thing reading the telemetry. And a machine cannot make up for missing structure by remembering the org chart.
Observability was never really about seeing. It was about understanding, and dashboards were a compressed format we mistook for the thing itself. The window was useful. But you cannot hand a window to an operator with no eyes. You have to hand it a model.
So here’s the test I’d run this week. Take your last incident’s timeline. For each fact a human supplied from memory — the deploy, the ownership, the past incident, the “oh, that’s downstream of the queue” — ask whether that fact exists anywhere a machine can query.
Every no on that list is a place your agents are currently guessing. Confidently, and in prose that reads exactly like an answer.
I lead AI transformation for a global SaaS platform, and the operational context contract above is the part of the work with the least glamour and the highest return. If you’ve built unified change timelines or structured runbooks at scale, I want to know what broke first. Find me on X, LinkedIn, or Telegram.