Observability Copilot

1. 01An anomaly appears somewhere in a Kubernetes estate running many applications, across many namespaces and environments.
2. 02Find the right logs in Grafana/Loki — if you know which application and namespace.
3. 03Hand-craft the correct PromQL to see the metric you need.
4. 04Dig through Jaeger for the trace.
5. 05Check endpoint health one by one.

A single agent on top of a custom MCP server I wrote for the Grafana stack. For metrics it has three core tools — list_metrics to discover what exists, get_label_values to find the right application, namespace and pod, and execute_promql to run a query. With those, the agent writes its own PromQL for whatever was asked rather than choosing from canned queries, runs it, refines it if the result is wrong, and renders the graph itself. The same server reaches Loki logs, Jaeger traces and a live endpoint-health snapshot.

Give any engineer the full observability stack in plain English — without memorising PromQL or the shape of the estate.

Capabilities

• 01Metric discovery — list_metrics
• 02Label discovery for app, namespace and pod — get_label_values
• 03Custom PromQL written by the agent, run via execute_promql
• 04Graphs rendered by the agent from the query result
• 05Loki log retrieval and summarisation · Jaeger traces
• 06Live platform snapshot — every endpoint tested on demand

1. in

   Trigger

   An engineer asks a question in plain language — “how has memory on this service trended over the last hour?”.

2. 01

   list_metrics — find which metric answers the question

3. 02

   get_label_values — resolve the application, namespace and pod

4. 03

   Write a PromQL query for exactly that question

5. 04

   execute_promql — run it; if the result is empty or wrong, refine and run again

6. 05

   Render the result as a graph, with the query shown

7. out

   Output

   The answer, the graph and the PromQL that produced it — or a log summary, a trace or an endpoint snapshot when that is what the question needed.

Components

• Chat / query interface
• Observability agent — Claude Sonnet 4.5, single-agent design
• Custom Grafana-stack MCP server (built for this) + official Grafana MCP server
• Loki (logs) · Prometheus (metrics) · Jaeger (traces)
• Live endpoint health checker
• Kubernetes estate — multi-application, multi-environment

Model

Claude Sonnet 4.5

Tooling

Custom Grafana-stack MCP serverOfficial Grafana MCP server

Observability

GrafanaLokiPrometheusJaeger

Platform

Kubernetes

• 01

  Generating PromQL precise enough to avoid high-cardinality timeouts across a large multi-application estate — the naive query is often the expensive one.

• 02

  Resolving the correct application, namespace and pod from a question an engineer typed in a hurry.

• 03

  Filtering repetitive, noisy log lines out of a summary without dropping the one line that mattered.

• Engineers reach the right logs, metrics and traces without knowing PromQL or the topology of the estate.

• Manual log searching, manual PromQL authoring and one-by-one endpoint checks are removed from the diagnosis path.

• Running in enterprise production across a Kubernetes estate hosting multiple applications and environments.

Written as a de-identified case study. The employer's identity, internal cluster and application names, dashboards, actual log contents and usage figures are not disclosed. What is described here is the architecture pattern and the capability — nothing proprietary to the environment it runs in.