Datasets overview
Searchable document collections your agents and workflows can use — what a dataset holds, and how to find your way around one.
A dataset is a searchable collection of your documents. You fill it by uploading files or by pointing it at a folder in your storage, and Compass reads each document, splits it into passages, and indexes them for search by meaning. Then your agents, workflows, and Compass Chat can search it and answer from your own material, naming the document each passage came from.
Datasets live under Studio → Datasets. Each one belongs to the project it's created in.
What a dataset holds
- Documents — each file you add becomes a document. Compass reads its text, splits it into chunks, and turns each chunk into a vector with the dataset's embedding model. A document only appears once it's fully indexed, so a file that fails never shows up half-there.
- Originals — the original file behind every uploaded or synced document is kept with the dataset, so it can be read again when you change how documents are processed.
- Sources — folders in a storage connection the dataset keeps itself in sync with, on a schedule. See Sources & sync.
- A pipeline — how each file is turned into searchable chunks: how it's read, whether scanned pages use OCR, and how text is split. See Pipelines.
The embedding model is fixed
Every dataset is built with one embedding model, chosen when you create it. It can't be changed afterwards: the index only works with the model that built it, and searches are embedded with the same model so they match. To use a different model, create a new dataset with it.
The Datasets page
Studio → Datasets lists the project's datasets with their status, number of sources, pipeline, embedding model, and when they last changed. Search by name or description, click New Dataset to create one, or click a row to open it.
Inside a dataset
A dataset's page shows its document and chunk counts, its pipeline, and its embedding model, across these tabs:
| Tab | What's there |
|---|---|
| Documents | Every indexed document with its status, source, chunk count, and when it was indexed. Upload files here, and remove documents you no longer want. |
| Sources | The folders the dataset syncs from, each with its schedule, last sync, and a Sync now button — plus an Uploads card for files added by hand. |
| Pipeline | How documents are processed, and a Re-ingest button to apply new settings to what's already there. The embedding model is shown here, marked Fixed. |
| Runs | Recent ingestions and syncs (the latest 200), with progress, status, and errors. View opens a run in Observability. |
| Settings | The dataset's name and description, and Delete dataset. Admins and Developers only. |
Status
A dataset's status badge tells you where it stands:
- Ingesting — documents are being processed right now.
- Has failures — a document's latest ingestion failed. The Runs tab shows why.
- Not in source — some documents were removed from their source but are still in the dataset. See removed files.
- Empty — no documents yet.
- Up to date — everything is indexed.
Using a dataset
Once a dataset has documents, you can search it from almost anywhere:
- In Compass Chat — add it from the Tools menu, or just ask; chat can list and search your datasets on its own.
- On the canvas — the Dataset component searches a dataset as a workflow step, or as a tool you give an agent.
- From your code — the API, SDK, and CLI can search, upload, and manage datasets.
See Using datasets.
Who can do what
Admins and Developers can create datasets, add documents and sources, and change settings. Reviewers can open datasets, and search them through the API, CLI, or SDK, but not change them.
Deleting a dataset
Settings → Delete dataset permanently removes the dataset with every document, chunk, and source in it. A dataset can't be deleted while a workflow or agent uses it — remove it from them first.