Observability Copilot
Natural-language access to logs, metrics, traces and live endpoint health across a multi-application Kubernetes estate — built on a custom Grafana-stack MCP server I wrote because the one I needed did not exist.
Engagement
Anonymized — Fortune 500 pharmaceutical enterprise
0
lines of PromQL an engineer has to write
The problem
What this replaced
The manual process, before
- 01An anomaly appears somewhere in a Kubernetes estate running many applications, across many namespaces and environments.
- 02Find the right logs in Grafana/Loki — if you know which application and namespace.
- 03Hand-craft the correct PromQL to see the metric you need.
- 04Dig through Jaeger for the trace.
- 05Check endpoint health one by one.
Who has this problem
DevOps engineers, SREs and infrastructure operators.
Why it costs money
Diagnosis speed is gated on how well an individual engineer knows PromQL and the topology of the estate. Slow diagnosis extends outages, risks SLA breach, and costs money by the minute.
The build
What I built
A single agent with an observability toolkit. It resolves the relevant application, namespace and pod from a plain-language question, then fetches and summarizes logs, generates and graphs PromQL, retrieves traces, or takes a live health snapshot across every endpoint.
Give any engineer instant access to the full observability stack in plain English — without memorizing PromQL or the shape of the estate.
Capabilities
- Dynamic application / namespace / pod discovery from a natural-language query
- Raw log retrieval plus summarization
- Natural language → PromQL generation, executed and graphed
- Jaeger trace retrieval
- Live platform snapshot — tests every endpoint on demand
How it runs
The workflow
Trigger
An engineer asks a question in plain language.
- 01Agent interprets the query and selects the right tool(s) from its toolkit
- 02Discovers the specific application, namespace and pod where relevant
- 03Calls log fetch, PromQL query, trace fetch or endpoint snapshot — chaining calls when one is not enough
- 04Formats the result for the shape of the question asked
Output
A summarized log, a PromQL query with its rendered graph, a Jaeger trace, or a live endpoint health snapshot — whichever the question called for.
Architecture
How it fits together
Components
- Chat / query interface
- Observability agent — Claude Sonnet 4.5, single-agent design
- Custom Grafana-stack MCP server (built for this) + official Grafana MCP server
- Loki (logs) · Prometheus (metrics) · Jaeger (traces)
- Live endpoint health checker
- Kubernetes estate — multi-application, multi-environment
Data flow
01
Query → agent selects tool(s)
02
Discover app / namespace / pod
03
Fetch logs, metrics, traces or endpoint health via MCP
04
Format and return — summary, graph, trace or snapshot
The guardrail
What it is not allowed to do
Read-only by construction.
The agent retrieves and visualizes. It does not act on infrastructure. An SRE reviews what comes back and decides what to do about it. It was deliberately built as a single agent with a tight toolkit rather than a multi-agent pipeline — the extra failure surface of a fan-out architecture was not worth it for a system whose job is to answer questions accurately under incident pressure.
Stack
What it is built on
- Model
- Claude Sonnet 4.5
- Tooling
- Custom Grafana-stack MCP serverOfficial Grafana MCP server
- Observability
- GrafanaLokiPrometheusJaeger
- Platform
- Kubernetes
Hard parts
What made it difficult
- 01
Generating PromQL precise enough to avoid high-cardinality timeouts across a large multi-application estate — the naive query is often the expensive one.
- 02
Resolving the correct application, namespace and pod from a question an engineer typed in a hurry.
- 03
Filtering repetitive, noisy log lines out of a summary without dropping the one line that mattered.
Outcome
What changed
Top 10
most-used agents in the department, by active users
Ranks among the ten most actively used agents in the department — adoption, not just deployment.
Engineers reach the right logs, metrics and traces without knowing PromQL or the topology of the estate.
Manual log searching, manual PromQL authoring and one-by-one endpoint checks are removed from the diagnosis path.
Running in enterprise production across a Kubernetes estate hosting multiple applications and environments.
Written as a de-identified case study. The client's identity, internal cluster and application names, dashboards and actual log contents are not disclosed. What is described here is the architecture pattern and the capability — nothing proprietary to the environment it runs in.
Limited availability
Tell me the problem.
I’ll tell you if an agent is the answer.
A 30-minute call, no pitch. Describe the manual loop that hurts and I’ll give you a straight read on whether this is worth building — including when the honest answer is a script, not an agent.
30 minutes · video call · no pitch