Freshness Monitoring

Set SLAs on data freshness and detect stale data before it impacts your business

Freshness monitoring tracks when your data was last updated and alerts you when it becomes stale. Stale data can be just as damaging as wrong data. If your dashboard shows yesterday’s numbers, decisions made today could be wrong.

Prerequisites: Before setting up freshness monitoring, you need:

• A connected data source with discovery completed
• Assets with timestamp columns (created_at, updated_at, or similar)

Why Freshness Matters

Data has an expected update cadence. When that cadence breaks, something is wrong:

How Freshness Monitoring Works

AnomalyArmor tracks freshness by monitoring timestamp columns:

1. You specify which column indicates “when data was updated”
2. Discovery queries the maximum value of that column
3. We compare against your configured SLA
4. Alert fires if data is older than allowed

-- What AnomalyArmor checks
SELECT MAX(created_at) FROM orders;
-- Result: 2024-01-15 07:58:32

-- Your SLA: Updated within 1 hour
-- Current time: 2024-01-15 08:30:00
-- Data age: 32 minutes
-- Status: OK Within SLA

Setting Up Freshness Monitoring

1. Navigate to Asset

   Go to Assets and click on the table you want to monitor.

2. Open Freshness Tab

   Click the Freshness tab in the asset details.

3. Select Timestamp Column

   Choose the column that best represents data recency:

   Column Type	Best For
   created_at	Append-only tables (events, logs)
   updated_at	Tables with updates (user profiles)
   loaded_at	ETL destination tables
   _etl_timestamp	Warehouse staging tables

   Choose a column that updates when new data arrives. For tables with updates, use updated_at instead of created_at to track the most recent changes.

4. Set Expected Frequency

   Define how often you expect new data:

   • Every 15 minutes
   • Hourly
   • Every 6 hours
   • Daily
   • Weekly
   • Custom (specify hours)
5. Configure Alert Threshold

   Set when to trigger alerts:

   Alert Configuration:

   • Warning threshold: 80% of expected (optional)
   • Violation threshold: 100%+ of expected

   Example (hourly expectation):

   • Warning: Data is 48+ minutes old
   • Violation: Data is 60+ minutes old
6. Save Configuration

   Click Save to activate freshness monitoring.

SLA Strategies

Start Conservative

Set SLAs with buffer room to avoid false positives:

Actual Cadence	Recommended SLA	Why
Every hour	2 hours	Allows for occasional delays
Every day	36 hours	Accounts for timing variations
Every 15 min	30 minutes	Double the expected window

Start lenient and tighten over time. It’s easier to make SLAs stricter than to deal with alert fatigue from SLAs that are too tight.

By Data Criticality

Criticality	SLA Approach	Example
Critical	Expected + 25%	Revenue: 1hr expected → 75min SLA
High	Expected + 50%	Orders: 1hr expected → 90min SLA
Medium	Expected × 2	Analytics: 1hr expected → 2hr SLA
Low	Expected × 3	Reports: daily expected → 3 day SLA

By Update Pattern

Streaming/Real-time

For data expected within minutes:

Setting	Value
Expected	Every 5 minutes
Warning	10 minutes
Violation	15 minutes
Alert	Slack + PagerDuty

Batch/Hourly

For hourly ETL jobs:

Setting	Value
Expected	Every hour
Warning	80 minutes
Violation	2 hours
Alert	Slack

Daily

For daily batch loads:

Setting	Value
Expected	Daily by 6am
Warning	8am (2hr buffer)
Violation	12pm (6hr buffer)
Alert	Email

Freshness Alerts

Setting Up Alert Rules

Create rules to notify you of freshness violations:

Field	Value
Rule	Critical Table Freshness
Event	Freshness Violation
Assets	orders, payments, users
Destinations	Slack, PagerDuty

Alert Content

Freshness alerts include:

[!] Freshness Violation

Asset: production.public.orders
Column: created_at

Expected: Updated every 1 hour
Last update: 3 hours 15 minutes ago
Data age: 3h 15m (SLA: 1h)

Detected: January 15, 2024 at 8:00 AM UTC

[View Asset] [View Dashboard]

Handling Special Cases

Weekends and Holidays

Some data doesn’t update on weekends:

Options:

1. Longer weekend SLA: Set different thresholds for Saturday/Sunday
2. Pause monitoring: Temporarily disable freshness checks
3. Adjust expectations: Set SLA to “72 hours” to cover full weekends

Maintenance Windows

During planned maintenance:

1. Disable rules: Toggle OFF freshness alert rules for affected assets
2. Document: Note expected staleness
3. Re-enable: Toggle rules back ON after maintenance

Infrequently Updated Tables

Some tables legitimately update rarely:

Table Type	Update Pattern	SLA Approach
Reference/lookup	Monthly	45-day SLA or no monitoring
Historical archive	Never	Don’t monitor freshness
Dimension tables	Daily/weekly	Match actual pattern

Viewing Freshness Status

Asset List View

In the Assets list, freshness status appears as indicators:

Indicator	Meaning
Green	Within SLA
Yellow	Warning threshold reached
Red	SLA violated
Gray	Freshness not configured

Asset Detail View

Click an asset to see:

• Current freshness status
• Last update timestamp
• Freshness history over time
• SLA configuration

The detail view shows a status card with the current freshness state, last update timestamp, data age, and a progress bar indicating how much of your SLA window has been consumed. Below that, a 7-day history shows freshness status over time.

Example freshness status:

Field	Value
Status	✓ Within SLA
Timestamp Column	created_at
Last Update	January 15, 2024 at 2:15 PM UTC
Data Age	45 minutes
SLA Threshold	2 hours
Progress	37% of SLA (45 min / 2 hr)

Freshness History: All checks in the last 7 days were within SLA.

Troubleshooting

False positive alerts

Problem: Freshness alerts fire when data is actually fine.

Solutions:

1. Verify timestamp column choice. Is it the right one?
2. Loosen SLA threshold
3. Check timezone handling (UTC vs. local)
4. Review discovery schedule vs. SLA timing

Missed violations

Problem: Data was stale but no alert fired.

Solutions:

1. Verify freshness monitoring is enabled for the asset
2. Check alert rule configuration
3. Confirm discovery is running frequently enough
4. Verify timestamp column has recent values

Wrong timestamp column

Problem: Freshness showing incorrect values.

Solutions:

1. Review column choice
2. For tables with updates, use updated_at not created_at
3. For ETL tables, use the load timestamp column
4. Ensure column is always populated (no NULLs)

Timezone issues

Problem: Freshness calculations seem off by hours.

Solutions:

1. Check timestamp column timezone
2. AnomalyArmor normalizes to UTC
3. Ensure consistent timezone handling in your ETL

Best Practices

1. Start with Critical Tables

Focus monitoring on:

• Revenue-impacting tables
• Customer-facing dashboard sources
• Compliance-required data

2. Align with Business Needs

Ask: “When would stale data cause a problem?”

Scenario	Acceptable Staleness	SLA
Real-time dashboard	Minutes	15 min
Daily executive report	Hours	Same-day
Monthly compliance	Days	1 week

3. Coordinate with ETL Schedules

Phase	Time
ETL Schedule	2:00 AM daily
Processing time	~30 minutes
Data available	~2:30 AM
SLA	4:00 AM (90 min buffer)

4. Use Warning Thresholds

Configure two levels:

1. Warning: “Heads up, getting stale” → Slack
2. Violation: “SLA breached” → PagerDuty

What’s Next

Set Up Metrics

Track row counts, null percentages, and detect anomalies

Configure Alerts

Get notified when freshness SLAs are violated

Report Badges

Embed freshness status in dashboards

Best Practices

Reduce alert fatigue