PagerDuty

Send AnomalyArmor alerts to PagerDuty for incident management

Send AnomalyArmor alerts to PagerDuty to trigger incidents and page on-call engineers. Critical schema changes or freshness violations can automatically create PagerDuty incidents for immediate response.

Why PagerDuty?

PagerDuty is ideal for alerts that require immediate human intervention:

• On-call routing: Automatically page the right person based on schedules
• Escalation: Ensure critical alerts don’t go unnoticed
• Incident management: Track response and resolution
• Deduplication: Related alerts are grouped together

Prerequisites

Before you begin:

• PagerDuty account with admin access (or permission to create integrations)
• AnomalyArmor account with alert configuration permissions
• A PagerDuty service configured for your data infrastructure

Setup Guide

Step 1: Create a PagerDuty Integration

In PagerDuty:

1. Go to Services and select (or create) a service for data alerts
2. Click Integrations tab
3. Click Add Integration
4. Search for Events API v2 and select it
5. Click Add
6. Copy the Integration Key (also called Routing Key)

The Integration Key is a 32-character string that looks like: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

Step 2: Add Destination in AnomalyArmor

1. Log in to AnomalyArmor
2. Click Alerts in the left sidebar
3. Select Destinations tab
4. Click Add Destination
5. Select PagerDuty

Step 3: Configure the Destination

Enter the following:

Field	Description
Name	A descriptive name (e.g., “PagerDuty - Data Platform”)
Integration Key	The key copied from PagerDuty

Step 4: Test the Connection

Click Test to send a test event to PagerDuty.

Success! Test event sent to PagerDuty
Dedup Key: anomalyarmor-test-12345

Check PagerDuty to confirm the incident was created.

Remember to resolve the test incident in PagerDuty to avoid confusion.

Step 5: Save

Click Create Destination to complete the setup.

Alert Format

AnomalyArmor sends events using the PagerDuty Events API v2 format:

Field	Value
Summary	Alert title with context
Source	anomalyarmor
Severity	critical (all PagerDuty alerts are high priority)
Dedup Key	Unique identifier to group related alerts

Since PagerDuty is reserved for critical alerts, all events sent to PagerDuty use the critical severity level. Use Slack or email for lower-priority notifications.

Custom Details

Each PagerDuty incident includes:

• Rule name that triggered the alert
• Event type (schema_change, freshness_violation, etc.)
• Alert description
• Asset ID
• Link to view in AnomalyArmor

Best Practices

Use PagerDuty for Critical Alerts Only

Don’t route all alerts to PagerDuty. Reserve it for events that require immediate action - typically production schema changes that could break pipelines.

Good use cases:

• Production column removed
• Critical table freshness SLA violated
• Breaking schema changes in production

Better handled elsewhere:

• Development database changes (use Slack)
• Informational schema additions (use email)
• Routine freshness warnings (use Slack digest)

Set Up Proper Escalation

Configure your PagerDuty service with:

1. Acknowledgement timeout: Auto-escalate if not acknowledged (e.g., 5 minutes)
2. Escalation policy: Include backup responders
3. On-call schedule: Ensure 24/7 coverage for critical services

Combine with Other Destinations

Create alert rules that send to multiple destinations:

Production Breaking Changes

• Event: Schema Change
• Scope: production databases
• Conditions: Column removed OR type changed
• Destinations: PagerDuty (immediate paging), Slack #data-incidents (team visibility), Email [e-mail] (record)

Troubleshooting

”Invalid routing key”

Cause: The integration key is incorrect or the integration was deleted in PagerDuty.

Fix:

1. Go to your PagerDuty service
2. Check the Events API v2 integration still exists
3. Copy a fresh integration key
4. Update the destination in AnomalyArmor

Incidents not appearing

Cause: Service is disabled or in maintenance mode.

Fix:

1. Check the PagerDuty service is enabled
2. Verify no maintenance window is active
3. Check the Events API v2 integration is active
4. Use the Test button to verify connectivity

Duplicate incidents

Cause: Multiple alert rules triggering for the same event.

Fix: AnomalyArmor includes a deduplication key with each event. PagerDuty will group alerts with the same dedup key into a single incident. If you’re seeing duplicates:

1. Review your alert rules for overlap
2. Consider combining rules or adjusting scopes

Rate limiting

Cause: Too many events sent in a short period.

Fix:

1. Review alert rule thresholds
2. Consider using alert grouping/digest for high-volume events
3. Route lower-priority alerts to Slack or email instead

Security

Data Sent to PagerDuty

Alert events contain:

• Asset names (database, schema, table names)
• Change types and descriptions
• Timestamps
• Rule information

Alert events do not contain:

• Actual data values
• Database credentials
• Connection strings
• Query results

Revoking Access

To disconnect AnomalyArmor from PagerDuty:

1. In AnomalyArmor: Delete the PagerDuty destination
2. In PagerDuty: Remove the Events API v2 integration from the service

Next Steps

Alert Rules

Create rules that route to PagerDuty

Best Practices

Reduce alert fatigue and page only when necessary