Upload Lineage Data

Import data lineage from dbt manifests, dbt Cloud, or manual definitions

AnomalyArmor can visualize how your data flows from source to destination, but it needs lineage data to work with. This guide covers three ways to get lineage into AnomalyArmor:

1. Upload a dbt manifest.json file (most common)
2. Sync from dbt Cloud automatically
3. Define lineage manually via the API

Option 1: Upload a dbt manifest.json

If you use dbt, the fastest way to populate lineage is uploading your manifest.json file. This file contains your full DAG, including all models, sources, seeds, and their dependencies.

Generate the manifest

Run one of these dbt commands to produce target/manifest.json:

# Either of these generates a manifest.json in target/
dbt parse      # Fastest, parses without compiling
dbt compile    # Compiles SQL, also generates manifest
dbt run        # Full run, also generates manifest

Upload via the API

cURL

curl -X POST \
  "https://api.anomalyarmor.ai/api/v1/assets/{asset_id}/lineage/upload" \
  -H "Authorization: Bearer aa_live_xxx" \
  -F "file=@target/manifest.json" \
  -F "sync_to_catalog=true"

What gets imported

AnomalyArmor parses the nodes and parent_map from your manifest to extract:

• Models (transformations in your dbt project)
• Sources (raw tables dbt reads from)
• Seeds (CSV files loaded by dbt)
• Parent-child relationships between all of the above

The sync_to_catalog parameter (default: true) also triggers a schema discovery job so your dbt models appear as assets in the catalog.

Response

{
  "data": {
    "asset_id": "550e8400-e29b-41d4-a716-446655440000",
    "sync_stats": {
      "nodes_created": 42,
      "nodes_updated": 8,
      "edges_created": 67,
      "edges_updated": 3
    },
    "manifest_metadata": {
      "generated_at": "2025-03-15T10:30:00Z",
      "dbt_version": "1.7.4",
      "project_name": "my_analytics"
    },
    "catalog_sync_job_id": "job-uuid-here"
  }
}

Automate uploads in CI/CD

Add a manifest upload step after your dbt run completes:

# .github/workflows/dbt.yml
jobs:
  dbt-run:
    steps:
      - name: Run dbt
        run: dbt run

      - name: Upload lineage to AnomalyArmor
        run: |
          curl -X POST \
            "https://api.anomalyarmor.ai/api/v1/assets/$ASSET_ID/lineage/upload" \
            -H "Authorization: Bearer $ARMOR_API_KEY" \
            -F "file=@target/manifest.json"
        env:
          ARMOR_API_KEY: ${{ secrets.ARMOR_API_KEY }}
          ASSET_ID: ${{ vars.ARMOR_ASSET_ID }}

Option 2: Sync from dbt Cloud

If you use dbt Cloud, AnomalyArmor can fetch the manifest directly from your dbt Cloud account. No file upload needed.

cURL

curl -X POST \
  "https://api.anomalyarmor.ai/api/v1/assets/{asset_id}/lineage/dbt-cloud/sync" \
  -H "Authorization: Bearer aa_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "account_id": "12345",
    "api_token": "dbtc_your_token_here",
    "job_id": "67890"
  }'

Finding your dbt Cloud credentials

Parameter	Where to find it
account_id	dbt Cloud URL: cloud.getdbt.com/deploy/**12345**/...
api_token	dbt Cloud > Account Settings > API Access > Service tokens
job_id	dbt Cloud > Jobs > select your job > ID in the URL

Use a service token with at least the “Read artifacts” permission. Personal tokens work but are tied to individual users.

Option 3: Define lineage manually

For data sources that are not managed by dbt, you can define lineage nodes and edges directly via the API.

Create a lineage node

curl -X POST \
  "https://api.anomalyarmor.ai/api/v1/assets/{asset_id}/lineage/nodes" \
  -H "Authorization: Bearer aa_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "unique_id": "source.crm.customers",
    "name": "customers",
    "resource_type": "source",
    "schema": "crm",
    "database": "production"
  }'

Create a lineage edge

curl -X POST \
  "https://api.anomalyarmor.ai/api/v1/assets/{asset_id}/lineage/edges" \
  -H "Authorization: Bearer aa_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "parent_unique_id": "source.crm.customers",
    "child_unique_id": "model.analytics.dim_customers",
    "relationship_type": "derives_from"
  }'

Updating lineage

When your dbt project changes, re-upload the manifest. AnomalyArmor handles updates intelligently:

• New nodes and edges are created
• Existing nodes are updated with new metadata
• Relationships that no longer exist are removed

For a clean reset, delete all lineage from a source first:

curl -X DELETE \
  "https://api.anomalyarmor.ai/api/v1/assets/{asset_id}/lineage/source/dbt" \
  -H "Authorization: Bearer aa_live_xxx"

Limits

Constraint	Value
Max manifest file size	50 MB
File format	JSON (UTF-8 encoded)
Required manifest keys	nodes, parent_map

Next steps

Query Lineage

Explore upstream and downstream dependencies

Impact Analysis

Check downstream impact before schema changes

dbt Integration

Add quality gates to your dbt workflows

AI Agent: Lineage

Ask natural language questions about data flow