Sources & sync
Keep a dataset in step with a folder in your storage — on a schedule, picking up new and changed files, and flagging removed ones.
A source is a folder in a storage connection that a dataset keeps itself in sync with. Add one, and the dataset ingests everything in the folder; after that, each sync picks up what's new or changed. It's how a dataset stays current without anyone uploading files by hand.
Today, sources come from S3 connections — AWS S3 or any S3-compatible store. A dataset can have as many sources as you like, each on its own schedule, alongside files you upload yourself.
Adding a source
Open the dataset's Sources tab and click Add source (or add sources while creating the dataset). Then fill in:
| Setting | What it does |
|---|---|
| Connection | The storage connection to read from. It can't be changed later — to switch, add a new source. |
| Folder | Browse to the folder to sync. The browser shows how many files are there. |
| Include subfolders | On by default. Off syncs only the files directly in the folder. |
| Name | Optional. Left blank, the source is named after its connection and folder. |
| File types | Optional. A comma-separated list of extensions, such as pdf, docx. Leave it blank to sync every file. |
| Keep in sync | On by default: the source syncs on a schedule. Off means it syncs only when you click Sync now. |
| When a file is removed from the source | Keep and flag it, or remove it — see below. |
Click Add & sync. The first sync starts straight away. (While you're creating a dataset, the button reads Add source, and the first sync starts when you click Create dataset.)
Filter to the files you want
With no File types filter, every file in the folder is sent through the pipeline — and a file the pipeline can't read fails its run and marks the dataset Has failures. If the folder holds more than documents, list the types you want.
Schedules
With Keep in sync on, choose how often the source syncs:
- Hourly — at a chosen minute past each hour.
- Daily — at a chosen time.
- Weekly — at a chosen time, on the days you pick (Monday to Friday by default).
- Monthly — at a chosen time, on a chosen day of the month.
A plain-English summary shows exactly when it will run. Times are in UTC. (The API and CLI can set any time zone.)
If a sync is still running when the next one is due, the next one is skipped. Syncs missed while Compass was unavailable aren't made up later.
To sync right away, click Sync now on the source.
What a sync does
Each sync lists the folder and compares it with what the dataset already holds:
- New files — a file whose path the dataset hasn't seen — are read and indexed.
- Changed files — a file whose content changed since it was indexed — are read again, and their chunks replaced. The document keeps its identity; it isn't duplicated.
- Unchanged files are left alone.
- Files already being ingested are left to finish.
Each new or changed file is its own run on the Runs tab. A file that fails isn't recorded, so the next sync tries it again.
A renamed or moved file counts as new at its new path, and removed at its old one.
When a file is removed
What happens when a file disappears from its source is a per-source choice:
- Keep it and flag it (the default) — the document stays searchable but is marked Not in source and no longer updates. The dataset shows how many documents are in this state; the Documents tab can filter to them, and Remove all clears them in one step. If the file comes back, the flag is cleared on the next sync.
- Remove it from the dataset — the document and its chunks are deleted on the next sync.
Keeping is the safe default: a source folder that's reorganized or briefly emptied doesn't quietly wipe out what your agents can find.
Editing and removing sources
Choose Edit from a source's … menu to change its folder, filters, schedule, or removal behavior. Changes apply from the next sync. Pointing a source at a different folder means the old folder's documents are no longer found, so they're flagged or removed according to the source's setting.
Remove stops the source syncing and removes every document that came from it. Uploaded documents aren't affected.