Ingestion specs — worked examples

These are real specs (lightly trimmed) showing how detection, parsing, and promotion steps fit together. Refer back to the promotion-steps catalog for each step.

Example 1 — Polaris SAV archive

A .sav archive is a Polaris SQL dump. The spec extracts SQL tables, then runs one promotion per output dataset. Two are shown: brands and inventory_items.

{
  "spec_id": "polaris_sav",
  "spec_version": 1,
  "name": "Polaris SAV Archive",
  "pos_system": "polaris",
  "status": "ACTIVE",
  "scope": "GLOBAL",
  "is_default": true,
  "priority": 0,

  "detection": {
    "match_mode": "composite",
    "rules": [
      { "type": "data_source", "codes": ["pos_polaris"] },
      { "type": "filename_matches", "pattern": "(?i)\\.sav$|\\.polaris\\.sav$" }
    ]
  },

  "parsing": {
    "parser": {
      "type": "polaris_sav",
      "sql_filename": "0-full.sql",
      "extract_media": true,
      "tables": ["marque", "libelleproduit", "article", "detvte", "libtaille"]
    },
    "extra_fields": { "mode": "store_json", "target_column": "_extra" }
  },

  "tags": { "systems": ["polaris"], "formats": ["sav"] },

  "promotions": [
    {
      "dataset_id": "brands",
      "domain": "catalog",
      "source_table": "marque",
      "conformed_table": "brands",
      "merge_keys": ["organization_id", "code"],
      "steps": [
        { "type": "deduplicate", "columns": ["no_marque"] },
        { "type": "add_column", "column": "is_active", "expression": "true" },
        { "type": "inject_value", "column": "organization_id", "value_key": "organization_id" },
        { "type": "add_column", "column": "created_at", "expression": "current_timestamp()" },
        { "type": "rename_columns", "columns": { "no_marque": "code", "nom": "name", "no_pays": "country_id" } },
        { "type": "generate_id", "output_column": "id", "key_columns": ["code"] },
        { "type": "id_mapping_output", "mapping_type": "brands", "code_column": "code", "id_column": "id" }
      ]
    },
    {
      "dataset_id": "inventory_items",
      "domain": "inventory",
      "source_table": "libelleproduit",
      "conformed_table": "inventory_items",
      "merge_keys": ["organization_id", "code"],
      "source_tables": ["detvte", "article", "libtaille"],
      "steps": [
        { "type": "deduplicate", "columns": ["no_libelleproduit"] },
        { "type": "sequential_join", "target_dataset": "detvte", "join_column": "no_libelleproduit",
          "select_columns": ["no_libelleproduit", "no_critmod", "no_article"], "deduplicate_on": ["no_libelleproduit"] },
        { "type": "sequential_join", "target_dataset": "article", "join_column": "no_article",
          "select_columns": ["no_article", "no_libtaille", "no_magasin"], "deduplicate_on": ["no_article"] },
        { "type": "id_mapping_join", "mapping_type": "variant", "source_column": "no_critmod", "output_column": "variant_uuid" },
        { "type": "sequential_join", "target_dataset": "libtaille", "join_column": "no_libtaille",
          "select_columns": [], "select_expressions": { "_size_label": "libelle" }, "deduplicate_on": ["no_libtaille"] },
        { "type": "taxonomy_mapping", "taxonomy_type": "size", "source_column": "_size_label",
          "source_id_column": "no_libtaille", "output_column": "taxonomy_id" },
        { "type": "id_mapping_join", "mapping_type": "shop", "source_column": "no_magasin", "output_column": "shop_uuid" },
        { "type": "inject_value", "column": "organization_id", "value_key": "organization_id" },
        { "type": "rename_columns", "columns": { "no_libelleproduit": "code", "variant_uuid": "variant_id",
          "taxonomy_id": "taxonomy_size_id", "shop_uuid": "shop_id" } },
        { "type": "generate_id", "output_column": "id", "key_columns": ["code"] },
        { "type": "id_mapping_output", "mapping_type": "inventory_items", "code_column": "code", "id_column": "id" }
      ]
    }
  ]
}

What to notice

Detection gates on the Polaris data source and a .sav filename.
The brands promotion ends with generate_id then id_mapping_output — publishing a brands code→id mapping that other promotions can resolve.
The inventory_items promotion chains sequential_joins to gather fields across tables, resolves the variant and shop via id_mapping_join, maps the size label to a taxonomy id, and finally generates its own id.

Example 2 — Ginkoia tickets (Excel)

An Excel export of payment-tender lines. Note the section_context parser option and the update_columns enrichment promotion.

{
  "spec_id": "ginkoia_tickets",
  "spec_version": 2,
  "name": "Ginkoia — Tickets (Payment Tender Lines)",
  "pos_system": "ginkoia",
  "status": "ACTIVE",

  "detection": {
    "match_mode": "composite",
    "rules": [
      { "type": "data_source", "codes": ["pos_ginkoia", "user_uploads"] },
      { "type": "filename_matches", "pattern": "(?i)tickets" },
      { "type": "header_contains", "columns": ["Numéro", "Séquence", "Montant", "Mode paiement"] }
    ]
  },

  "parsing": {
    "parser": {
      "type": "excel",
      "header_row": 0,
      "strip_columns": [0, 1],
      "section_context": [
        { "label": "Magasin", "output_column": "_section_magasin" },
        { "label": "Poste",   "output_column": "_section_poste" }
      ]
    },
    "mapping": {
      "columns": {
        "Numéro":       { "target": "ticket_number",  "type": "STRING" },
        "Montant":      { "target": "amount",         "type": "DOUBLE" },
        "Mode paiement":{ "target": "payment_method", "type": "STRING" },
        "_section_magasin": { "target": "shop_name",  "type": "STRING" }
      }
    },
    "extra_fields": { "mode": "store_json", "target_column": "_extra_fields" },
    "validation": { "required_columns": ["ticket_number", "payment_method"], "min_rows": 1 }
  },

  "promotions": [
    {
      "dataset_id": "customers",
      "domain": "catalog",
      "conformed_table": "customers",
      "merge_keys": ["organization_id", "code"],
      "update_columns": ["postal_code", "updated_at"],
      "source_table": "Tickets",
      "steps": [
        { "type": "filter", "condition": "customer_postal_code IS NOT NULL AND customer_postal_code != ''" },
        { "type": "deduplicate", "columns": ["customer_name"] },
        { "type": "id_mapping_join", "mapping_type": "customers_by_name", "source_column": "customer_name", "output_column": "code" },
        { "type": "filter", "condition": "code IS NOT NULL" },
        { "type": "rename_columns", "columns": { "customer_postal_code": "postal_code" } },
        { "type": "inject_value", "column": "organization_id", "value_key": "organization_id" },
        { "type": "add_column", "column": "updated_at", "expression": "current_timestamp()" },
        { "type": "generate_id", "output_column": "id", "key_columns": ["organization_id", "code"] }
      ]
    }
  ]
}

What to notice

Detection combines a source-code gate, a filename regex, and a header_contains check on the expected columns.
section_context carries the “Magasin” / “Poste” section headers down onto each row.
The customers promotion is a targeted enrichment: update_columns limits the upsert to postal_code (and updated_at), so it backfills postal codes without overwriting the rest of an existing customer row. The two filter steps ensure only valid, resolvable rows are written.

Real seed specs live under scripts/db/seeds/seed-data/ingestion-specs/<system>/. New specs are validated at seed time (including the data_source codes).

API integration

API recipes

Platform & data model

Decision layer (reference)

Operations & infrastructure

Ingestion specs (reference)

Workflows (reference)

Tagging rules (reference)

Alerts (reference)

Ingestion specs — worked examples

Example 1 — Polaris SAV archive

Example 2 — Ginkoia tickets (Excel)

​Example 1 — Polaris SAV archive

​Example 2 — Ginkoia tickets (Excel)

Example 1 — Polaris SAV archive

Example 2 — Ginkoia tickets (Excel)