Datadog Anomaly RCA Agent

What it does

Datadog Anomaly RCA Agent is an SRE-focused MCP agent that detects metric anomalies in Datadog, correlates them with log error patterns, and generates a structured Root Cause Analysis report. The pipeline collects metric time series and logs from Datadog, detects anomaly windows via z-score analysis, extracts and correlates log error patterns, and synthesizes a narrative RCA via LLM (with a deterministic template fallback when no LLM is configured). It can optionally post the report to a Slack war-room channel and annotate a PagerDuty incident. Guardrails include a 24-hour query window cap, PII masking, and prompt-injection detection. Built for SRE and on-call teams who want a first-pass root-cause hypothesis in seconds instead of manually cross-referencing dashboards and logs.

Example prompts

• Run a full anomaly RCA for the checkout service over the last 6 hours
• Query metrics for the payments service and detect anomaly windows
• Correlate this anomaly with recent log errors and generate an RCA report
• Post the RCA report to Slack and add a note to the PagerDuty incident

Tools (10)

Tools the agent exposes — your AI client calls them automatically when it needs them.

• run_full — Run the full RCA pipeline end-to-end: collect metric + log data, detect anomalies, correlate with log errors, synthesize LLM narrative, return the structured RCA payload.
• query_metrics — Query Datadog metric time series for a service. Uses mock data unless DATADOG_LIVE=true and DD_API_KEY+DD_APP_KEY are set.
• query_logs — Query Datadog logs for a service in a time window. Uses mock data unless DATADOG_LIVE=true.
• correlate_anomaly — Detect anomaly windows in metric data and correlate them with log error patterns, with confidence scores.
• generate_rca — Generate a structured RCA JSON report (metric, anomaly_start, correlated_errors, probable_cause, confidence, recommended_actions).
• notify_slack — Post the RCA report text to a Slack channel. Requires SLACK_BOT_TOKEN for live operation.
• add_pagerduty_note — Add an RCA note to a PagerDuty incident. Requires PD_API_KEY for live operation.
• discover_intent — Discover what this agent can do: capabilities, credential slots, guardrails and a quick-start guide.
• list_capabilities — List the agent's available tools and a brief description of each.
• plan_inputs — Plan/brainstorm the inputs for a tool: returns questions, schema and a ready-to-edit example.