Why incident.io?
incident.io is ideal for teams that want structured incident management:- Incident lifecycle: Track incidents from detection to resolution
- Automatic creation: Alerts create incidents automatically
- Slack integration: Automatically create incident channels
- Post-mortems: Built-in tooling for learning from incidents
Prerequisites
Before you begin:- incident.io account with API access
- AnomalyArmor account with alert configuration permissions
- Permission to create API keys in incident.io
Setup Guide
Step 1: Create an incident.io API Key
In incident.io:- Go to Settings → API Keys
- Click Create API key
- Give it a descriptive name (e.g., “AnomalyArmor Integration”)
- Ensure it has the
incident.writepermission - Click Create
- Copy the API key (it won’t be shown again)
API keys start with
inc_ followed by a long alphanumeric string.Step 2: Get Your Closed Status ID (Optional)
To enable automatic incident closing when alerts are resolved in AnomalyArmor:- Go to Settings → Incident Lifecycle in incident.io
- Click on a status with category “Closed” (e.g., “Closed”, “Resolved”)
- Copy the UUID from the URL (e.g.,
01FCNDV6P870EA6S7TK1DSYDG0)
When configured, resolving or dismissing an alert in AnomalyArmor will automatically close the corresponding incident in incident.io.
Step 3: Add Destination in AnomalyArmor
- Log in to AnomalyArmor
- Click Alerts in the left sidebar
- Select Destinations tab
- Click Add Destination
- Select incident.io
Step 4: Configure the Destination
Enter the following:| Field | Description |
|---|---|
| Name | A descriptive name (e.g., “incident.io - Data Platform”) |
| API Key | The API key copied from incident.io |
| Closed Status ID | (Optional) UUID of the status to set when closing incidents automatically |
Step 5: Test the Connection
Click Test to create a test incident in incident.io.Step 6: Save
Click Create Destination to complete the setup.Alert Format
AnomalyArmor creates incidents using the incident.io API v2:| Field | Value |
|---|---|
| Name | [AnomalyArmor] Alert title |
| Summary | Alert description with AnomalyArmor context |
| Mode | standard (not retrospective) |
| Visibility | public |
Incident Summary
Each incident includes:- Alert description
- Source: AnomalyArmor
- Rule name that triggered the alert
- Event type (schema_change, freshness_violation, etc.)
- Asset ID
Idempotency
AnomalyArmor includes an idempotency key with each incident request. This prevents duplicate incidents if the same alert is processed multiple times.Incident Lifecycle
AnomalyArmor supports full incident lifecycle management:| AnomalyArmor Action | incident.io Effect |
|---|---|
| Alert triggered | New incident created |
| Alert resolved | Incident status updated to configured closed status |
| Alert dismissed | Incident status updated to configured closed status |
Automatic incident closing requires the Closed Status ID to be configured. If not set, incidents must be closed manually in incident.io.
Best Practices
Reserve for Critical Alerts
Good use cases:- Production column removed or renamed
- Critical table freshness SLA violated
- Breaking schema changes in production databases
- Development database changes (use Slack)
- Informational schema additions (use email)
- Routine freshness warnings (use Slack digest)
Combine with Other Destinations
Create alert rules that send to multiple destinations: Production Breaking Changes- Event: Schema Change
- Scope: production databases
- Conditions: Column removed OR type changed
- Destinations: incident.io (incident creation), Slack #data-incidents (team visibility), Email data-eng-list@company.com (record)
Troubleshooting
”No API key configured”
Cause: The API key field is empty. Fix:- Edit the destination in AnomalyArmor
- Enter your incident.io API key
- Save the destination
”HTTP 401: Unauthorized”
Cause: The API key is invalid or expired. Fix:- Go to incident.io Settings → API Keys
- Verify the key exists and hasn’t been revoked
- Create a new API key if needed
- Update the destination in AnomalyArmor
”HTTP 403: Forbidden”
Cause: The API key lacks required permissions. Fix:- Go to incident.io Settings → API Keys
- Verify the key has
incident.writepermission - Create a new key with correct permissions if needed
Incidents not appearing
Cause: Rate limiting or API issues. Fix:- Check incident.io status page for outages
- Use the Test button to verify connectivity
- Check the AnomalyArmor alert history for delivery errors
Security
Data Sent to incident.io
Incident data contains:- Asset names (database, schema, table names)
- Change types and descriptions
- Timestamps
- Rule information
- Actual data values
- Database credentials
- Connection strings
- Query results
Revoking Access
To disconnect AnomalyArmor from incident.io:- In AnomalyArmor: Delete the incident.io destination
- In incident.io: Revoke the API key in Settings → API Keys
Next Steps
Alert Rules
Create rules that route to incident.io
Best Practices
Reduce alert fatigue and create incidents only when necessary
