Set Up Completeness Monitoring

Track row counts with ML-powered anomaly detection

Now that you have discovery running and alerts configured, let’s set up completeness monitoring. This tracks row counts in your tables and alerts you when data volumes are abnormal.

What You’ll Accomplish

By the end of this step, you’ll have:

• Completeness monitoring on a critical table
• ML-based anomaly detection learning your data patterns
• Alerts when row counts deviate from expected values

Why Completeness Monitoring?

Row count anomalies catch real problems:

Anomaly	What It Means
Sudden drop	Failed ETL job, data loss, broken pipeline
Unexpected spike	Duplicate loads, runaway inserts
Missing data	Source system outage, extraction failure
Gradual decline	Upstream issues, filtering bugs

Set Up Monitoring

1. Navigate to Your Table

   1. Go to Assets from the main navigation
   2. Click on a table you want to monitor (pick one that receives regular data)
   3. Click the Completeness tab
2. Enable Monitoring

   Click Enable Completeness Monitoring to open the configuration.

3. Configure Settings

   Setting	Recommended Value	Why
   Mode	Auto-learn	ML learns your normal patterns
   Time Window	24 hours	Matches daily ETL schedules
   Check Interval	Hourly	Catches issues quickly
   Sensitivity	2 (Medium)	Balanced alerting

   Auto-learn mode builds a baseline over 7 days, then uses ML to predict expected row counts. No manual thresholds needed.

4. Save

   Click Save. Monitoring starts immediately and begins learning patterns.

Understanding the Learning Phase

After enabling, you’ll see a Learning status badge on the monitoring card. The chart will show data points being collected, with a message like “Building baseline - 3 of 7 data points collected.”

What happens during learning:

• Row counts are captured at your check interval
• ML model analyzes patterns (daily, weekly, hourly trends)
• After 7+ data points, predictions activate

After Learning Completes

Once the baseline is established:

• Expected range shown on the chart
• Anomalies highlighted when row count falls outside predictions
• Alerts fire to your configured destinations

Explicit Mode (Alternative)

If you know exactly what to expect, use explicit mode:

Setting	Example
Mode	Explicit
Minimum Rows	10,000
Maximum Rows	50,000

Alerts fire when row count falls outside your defined range.

Which Tables to Monitor

Start with tables that:

• Receive data regularly (daily, hourly)
• Are critical to downstream reporting
• Have predictable volume patterns

Good candidates:

• Fact tables (orders, events, transactions)
• Staging tables from ETL pipelines
• Aggregation tables

Skip for now:

• Dimension tables (change infrequently)
• Archive tables
• Temporary/scratch tables

Troubleshooting

Still in 'Learning' after a week

Check these:

1. Is the table receiving new data?
2. Is the check interval appropriate for your data frequency?
3. View the history tab to see if captures are running

Too many alerts

Reduce noise:

1. Increase sensitivity (3 or 4 = fewer alerts)
2. Adjust time window to match your data patterns
3. Consider if this table has irregular patterns

Not getting alerts when expected

Verify:

1. Learning phase is complete
2. Alert rules are configured for completeness events
3. Check the history tab to see detected anomalies

What’s Next

Column Metrics

Track null percentages, distinct counts, and more

Freshness Monitoring

Ensure tables are updated on schedule