> ## Documentation Index
> Fetch the complete documentation index at: https://docs.solya.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingestion (technical)

> The ingestion spec model, run statuses, and the sandbox validate-then-promote flow.

This is the technical companion to the [data layer](/en/data-layer/overview) guides.

<Note>
  For the **exhaustive, field-level** reference — every detection rule, parser option, and
  promotion step type with parameters and worked examples — see
  [Ingestion specs (reference)](/en/developers/ingestion-specs/overview).
</Note>

## Ingestion spec model

An ingestion spec describes how to turn an incoming file into Solya's model:

* **Detection** — how the file is matched: `FILENAME`, `COLUMNS`, `SHEET_NAME`,
  `FILENAME_MATCHES`, `HEADER_CONTAINS`, with match modes `ANY`, `ALL`, `COMPOSITE`.
* **Parsing** — how columns are read (field types, formats).
* **Promotion** — the transformation pipeline. Step types include `COERCE_TYPES`,
  `FILTER`, `DEDUPLICATE`, `ADD_COLUMN`, `RENAME_COLUMNS`, `SELECT_COLUMNS`,
  `DROP_COLUMNS`, `INJECT_VALUE`, `JOIN` / `SEQUENTIAL_JOIN` / `AGGREGATION_JOIN`,
  `UNPIVOT`, `TAXONOMY_MAPPING`, `ID_MAPPING_OUTPUT` / `ID_MAPPING_JOIN`, `GENERATE_ID`.
  Write modes: `MERGE`, `APPEND`, `OVERWRITE_PARTITION`.

A spec has a **status** (`DRAFT`, `ACTIVE`, `DEPRECATED`, `ARCHIVED`), a **scope**
(`GLOBAL` or `ORG`), a `priority`, and an `isDeployed` flag (only deployed specs apply to
live files). Specs are versioned.

```mermaid theme={null}
flowchart TB
  file["Incoming File"]
  detect["Detection<br/>(match spec)"]
  parse["Parsing<br/>(read columns)"]
  promote["Promotion<br/>(transform steps)"]
  table["Silver/Gold<br/>Target Table"]
  
  file --> detect
  detect --> parse
  parse --> promote
  promote --> table
```

## Ingestion runs

An ingestion run has status `PENDING → RUNNING → SUCCESS / FAILED`, a trigger type
(`MANUAL`, `SCHEDULED`, `API`), a link to the underlying Databricks run, and structured
**logs** at `DEBUG` / `INFO` / `WARN` / `ERROR` (capped at 5000 entries, 2000 chars per
message). It records `triggeredAt`, `startedAt`, and `completedAt`.

## Sandbox: validate then promote

The [sandbox](/en/data-platform/sandbox) splits ingestion into two tracked runs:

* **Sandbox ingest run** — parses and validates an uploaded file into temporary tables and
  emits **findings** with a **severity** (`error` / `warning` / `info`) and a **stage**
  (detect, parse, map, loader, strict scan, quarantine). Status
  `PENDING → RUNNING → SUCCESS / FAILED`.
* **Sandbox promote run** — copies validated data into silver. Same statuses, plus a
  terminal **SUCCESS-but-aborted** state (`stats.aborted = true`) when the ingest had
  errors and promotion-with-errors wasn't allowed.

```mermaid theme={null}
stateDiagram-v2
  [*] --> PENDING
  PENDING --> RUNNING
  RUNNING --> SUCCESS
  RUNNING --> FAILED
  SUCCESS --> [*]
  FAILED --> [*]
```

A **spec recommendation run** profiles an uploaded file and proposes a spec plus per-dataset
row previews (same status set).

## File ingestion records

Files dropped into the landing area are tracked with a status of `UPLOADED`, `INGESTED`,
`ERROR`, or `DELETED`, and a source of `APP` (UI upload) or `CLIENT_IMPORTER` (external
importer tool).

<Note>
  All runs are organization-scoped and observable from the UI and API, with the same
  structured-log model across ingestion, sandbox, tag-, and alert-evaluation runs.
</Note>