Pipelines
How a dataset turns each file into searchable chunks — reading, OCR, chunking, and re-ingesting with new settings.
A dataset's pipeline is the process every file goes through on its way into the index: read the document's text, split it into chunks, embed each chunk, and store it. You choose the pipeline and tune its settings when you create a dataset, and can change them later on the Pipeline tab.
Standard Document Ingestion
Compass ships one pipeline, Standard Document Ingestion — "Read a document's text, using OCR for scanned pages, then chunk and index it." It's the default for every dataset.
Its settings are grouped by step:
| Step | Setting | Default | What it does |
|---|---|---|---|
| Read Document | Always use OCR | Off | Read every page with OCR, not just the ones with no text layer. Better tables and reading order — and a charge for every page. |
| Read Document | OCR connection | None | The connection used to read scanned pages. Without one, a scanned page is an error. |
| Index | Chunk size | 1000 | The longest a chunk may be, in characters. |
| Index | Overlap | 200 | When a single paragraph is longer than a chunk, the characters repeated across each cut, so a sentence that straddles one is still findable. |
| Index | Split on headings | On | Start a new chunk at every Markdown heading, so a chunk holds one idea. |
Reading documents
The first step reads each file as text:
- PDF — each page is read from its text layer, as plain text. A page that's only an image — a scan — needs OCR, which returns Markdown with headings and tables.
- Word (
.docx) — headings and tables are kept, as Markdown. - Excel (
.xlsx) — each sheet becomes a heading and a table, using calculated values. - Text — Markdown, CSV, and plain text are read as they are.
- Images — read with OCR only.
Any other type fails its run, naming the file.
Scanned documents and OCR
OCR comes from a provider connection with the OCR capability — today, Azure Document Intelligence. With an OCR connection set, only the pages that need it are sent to OCR, unless Always use OCR is on.
Without one, a document with scanned pages fails rather than being indexed with gaps, and its run names the pages: "Pages 3, 4 of 'statement.pdf' are images, not text." That's deliberate — a search can't tell it's missing half a document, so Compass doesn't index half a document.
Chunking
Text is split at Markdown headings first (with Split on headings on), then paragraphs are packed into chunks up to the Chunk size. A single paragraph longer than that is cut, with Overlap characters repeated across each cut; chunks that break between paragraphs or at headings don't overlap. When the text has headings, each chunk remembers the one it sits under, and search results include it. (A PDF's text layer has no headings, so its chunks are split by paragraph alone.)
Smaller chunks give more precise matches; larger ones give the model more context per result. The defaults suit most prose.
Changing settings and re-ingesting
Change the pipeline or its settings on the Pipeline tab and click Save. New settings apply to documents ingested from then on — documents already in the dataset keep the chunks they were built with.
To rebuild everything with the new settings, click Re-ingest. Every document that came from an upload or a source goes through the pipeline again, each as its own run. Documents still being ingested are skipped.
Watching runs
Every file processed is a run on the dataset's Runs tab — showing its status, the step it's on, and any error. While runs are in flight, a progress bar counts documents indexed, running, and failed. Filter to Running or Failed, and click View to open a run's full trace in Observability.
Runs execute in the dataset's project, so project and organization spend limits are checked before each one starts.