Creating a dataset
Choose an embedding model, add documents from storage or by upload, and pick how they're processed.
Creating a dataset takes three decisions: which embedding model indexes it, where its documents come from, and which pipeline processes them. You make all three in one panel.
Before you start
You need a provider connection whose model is an embedding model — look for the Embedding badge under Studio → Connections. OpenAI and LM Studio both offer embedding models. To sync from storage, you'll also want an S3 connection.
Create it
Under Studio → Datasets, click New Dataset. Give it a Name — unique in the project — and optionally a Description, then work through three steps.
Embedding
Pick the Connection whose model will index the dataset. The panel shows the model and its vector size. This can't be changed once the dataset exists.
Documents (optional)
Add documents now, or later from the dataset's page:
- Add source — sync a folder from a storage connection, on a schedule. See Sources & sync for every setting.
- Upload files — drop files in, or browse for them.
Wait for uploads to finish before you continue.
Click Create dataset. Compass creates the dataset, starts the first sync of each source, and begins ingesting your uploads — then opens the dataset on its Runs tab so you can watch documents arrive.
Uploading documents
You can upload files while creating a dataset, or any time afterwards from its Documents tab: drop files onto Upload files, then click Ingest. Each file becomes a document and runs through the pipeline on its own.
- Accepted files — PDF, text files (plain text, Markdown, CSV), and images. Images and scanned PDFs are read with OCR, so the pipeline needs an OCR connection.
- Size — up to 20 MB per file.
- Word and Excel —
.docxand.xlsxfiles are read when they come from a synced source; they can't be uploaded directly.
Uploading the same file twice creates two documents. To replace a document, remove the old one first — or keep the file in a synced folder, where a changed file updates its document in place.
What happens next
Each document is its own run, visible on the dataset's Runs tab with its progress and, if it fails, the reason. A document shows up on the Documents tab once it's indexed. Runs are executions like any other, in the dataset's project: they appear in Observability, and project and organization spend limits apply to them.