> For the complete documentation index, see [llms.txt](https://help.daasity.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.daasity.com/core-concepts/data-integrations/custom-data-inputs/new-data-file-csv-delimited-xls.md). # \[NEW] Data File (CSV, Delimited, XLS) Data File integrations load CSV, delimited, and Excel files from your own storage into your Daasity warehouse on a recurring schedule. You set them up yourself: point at the files, load a sample to detect the columns, map them, decide how the data loads, and test which files match before anything runs. ### How Data File integrations are organized Two things work together: * A **Data Connection** is the parent. It stores the connection to wherever your files live — S3, SFTP, Daasity Storage, or an email webhook. * A **Data File integration** is a child of that connection. Each one loads a set of matching files into one warehouse table. One Data Connection can have as many Data File integrations as you need. ### **Finding your Data File integrations** On the Integrations page, scroll past the API-based integrations to the **Data Connections** section. The line beneath each card name tells you what's on it: a file connection shows how many **integrations** it has, while a database connection shows **table replicators** instead. Select a card to open it. {% hint style="info" %} The standalone **Data Connections** page is being retired — you'll see a notice on it. Manage your connections from the Integrations page instead. {% endhint %} ### Supported sources and file types Data File integrations load **CSV**, **delimited**, and **Excel (XLS)** files from: * Amazon S3 * SFTP — note this is SFTP, not FTP * Email Webhook * Daasity Storage ### Creating a Data File integration Choose **New Integration**, then your file type. Sections appear as you go — Build Field Mappings once you've set a connection and path, then the rest once a sample file has loaded. #### Step 1: General | Field | What to enter | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** (required) | A label that identifies what this integration extracts. | | **Data Connection** | The connection where your files live. Pick an existing Data Connection or create a new one. Archived Data Connections aren't listed. | | **File Path** | The path or prefix to your file(s) on the Data Connection. Files matching this path are picked up on each run. | | **Date Pattern** | If your filenames contain a date, choose its format so each run can extract the date from the filename. Leave it as **None** when filenames have no date. | | **Field Separator** (required) | Comma (,), Pipe, or Tab — match whatever your file uses. Defaults to Comma. | Once you pick a Data Connection, Daasity shows its connection details beneath the field, with a **View connection** link. {% hint style="info" %} A mismatched **Field Separator** is a common cause of every value landing in a single column. \ \ Field Separator appears on the CSV and Delimited forms only. {% endhint %} **Path syntax** Select **Path syntax help** under the path field for a reference table. The two wildcards behave differently: | You write | Date Pattern | Matches files like | | ----------------------------- | ------------ | ---------------------------------------------------------------------------- | | `exports/report_*.csv` | *none* | any file in `exports/` ending in `.csv` | | `exports/report_*.csv` | `%Y-%m-%d` | `exports/report_2025-05-21.csv` — `*` must contain a matching date | | `exports/report_{{date}}.csv` | `%Y-%m-%d` | `exports/report_2025-05-21.csv` — the date must appear exactly at `{{date}}` | `*` matches any characters. `{{date}}` is stricter: it pins the date to that exact position, with no surrounding characters. {% hint style="info" %} The path and Date Pattern fields don't appear for **Email Webhook** or **Daasity Storage** Data Connections. Those receive files rather than searching a remote path. {% endhint %}
*** #### Step 2: Build Field Mappings Daasity detects your columns by reading a sample file, so you never type them by hand. First set **how many rows are in the header row**. Getting this right matters — it tells the parser where the headers are and where the data starts. Then populate the mappings one of two ways: * **Load from Data Source** — pulls a real file from the path you entered. Use this, because it also proves your Data Connection and path are working. * **Upload from your computer** — reads a sample from your computer. Fine for defining structure, but it doesn't validate your path. The Build Field Mappings section doesn't appear until you've chosen a Data Connection and entered a File Path. Test Expression becomes available at the same time.
*** #### Step 3: Review your field mappings Once a sample loads, every detected column appears in a table. | Column | What it does | | --------------------- | ------------------------------------------------------------------------------------------------------------------- | | **#** | Row number | | **Include** | Tick to load this column | | **Source Field** | The column name exactly as it appears in the source file | | **Destination Field** | The column name to use in the warehouse table | | **Format** | The data type to store the value as. Choose a date or timestamp type and a date-format selector appears beneath it. | Daasity detects the column names but **not** their data types. Set the Format yourself for anything that isn't text. If two destination names end up identical, Daasity flags them and won't let you save until you rename one.
*** #### Step 4: Choose how the data loads Open **Data Loading** and pick one of three actions. | Action | What happens | Use it when | | ------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | **Full Data Replacement** | Deletes all existing records in the destination table before loading. | The source file contains the full dataset each time, or you can't define a primary key. | | **Update / Insert** | Keeps existing records, loads new ones, and updates records that already exist. | You're loading incrementally and need to capture updates to records already loaded. | | **Append Only** | Keeps existing records and loads new ones only. | You're loading incrementally and don't need to capture updates. | **Setting the sync key** The sync key is how Daasity decides whether an incoming row is new or an update to a row it already has. Select the column or columns that uniquely identify each row. Under **Select your replication key type**, choose either: * **Individual key — primary key**, then pick the single column that identifies a row, or * **Compound key — multiple columns**, then pick the columns that are unique in combination. **Current sync key** shows what's in effect, reading `(no columns selected)` until you choose. {% hint style="warning" %} **Update / Insert requires at least one sync key column.** It's optional for Full Data Replacement and Append Only. {% endhint %} {% hint style="info" %} Columns are always combined in alphabetical order, so the order you pick them in doesn't matter. {% endhint %} **Advanced** * **Do not remove duplicates** — load every row as it arrives instead of matching rows against a key. Use this for rollup-style files where repeating entries each need their own row. * **Lookback Window (days)** — how many days back to scan for files on each run. Leave blank to use the default of 2 days. {% hint style="info" %} The lookback window also governs history loads. Daasity steps through history in chunks matching your window rather than making one request per day, which meaningfully reduces SFTP and S3 listing calls on long backfills. {% endhint %}
*** #### Step 5: Set the destination | Field | What to enter | | ------------------------------------ | -------------------------------------------------- | | **Destination Schema** (required) | The schema in your warehouse to load the data into | | **Destination Tablename** (required) | The table in your warehouse to load the data into | Both are fixed once the integration is created. *** ### Test Expression **Test Expression** queries your Data Connection using your current path and Date Pattern and shows you exactly which files would match. It's available while you're setting the integration up and on its page afterward. 1. Enter your **File Path** and choose a **Date Pattern**. 2. Select **Test Expression**. 3. The dialog shows the **Path** and **Date Pattern** it will use. Optionally set a **Test Date** to narrow the search. With a Date Pattern set, this filters on the date in the filename; with Date Pattern set to None, it filters on file modification time instead.
4. Select **Run Test**. The dialog closes and a **Files Found** panel appears beneath the Test Expression button, showing the path and test date used, how many files matched, and each matching filename. If nothing matched or the connection couldn't be reached, you'll see the actual error instead.
{% hint style="success" %} Run this before saving. It catches wrong folder paths, mismatched date patterns, and permission problems while you can still fix them, instead of the next morning when the workflow runs. {% endhint %} {% hint style="info" %} Test Expression isn't available for Email Webhook or Daasity Storage Data Connections, which have no remote path to search. {% endhint %} ### File Processing Log The integration page includes a **File Processing Log** showing every file Daasity has attempted, so you can confirm what was actually picked up. | Column | What it shows | | ------------------ | ---------------------------------- | | **Filename** | The file processed | | **Processed At** | When it was processed | | **Status** | The outcome of the attempt | | **Rows Extracted** | How many rows came from that file | | **Workflow** | The workflow run that processed it | Search by filename, or filter by status: Completed, Failed, Started, Enqueued, or Invalidated. If nothing has run yet you'll see *No processing history found.* {% hint style="success" %} Check here first when data looks missing. It tells you whether the file arrived at all — then use **Test Expression** to confirm your path still matches. {% endhint %}
### Editing a Data File integration Open the integration and choose **Edit**. You can change: * **Name** * **Load Action** — how data is loaded into the destination table * **Lookback Window (days)** * **Do not remove duplicates** * **Header Row Number** — the line number of the header row. Use 0 if the file has no header. * **Sync Key** — the destination columns that uniquely identify each row Under **Sync Key**, every destination column is listed with an **In Key** checkbox, its name, and its type. Tick the columns you want and **Current sync key** updates to match. The destination schema and table are fixed at creation. {% hint style="warning" %} Changing the sync key changes how rows are matched. Rows already loaded were written under the old key, so they can no longer be matched against incoming rows — review the table after the next run. {% endhint %} {% hint style="info" %} Some parsing options are admin-only and appear under **Admin Knobs**. If you need one of these, email : * Use positional header * Format fields * Use filename date in sync key * Remove escape characters * Remove inner quotes * Substitution dictionary {% endhint %} ### **Deleting a Data File integration** Open the integration, then choose **Delete**. A confirmation dialog asks what should happen to the data already loaded. | Option | What happens | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Keep the data | The integration is removed. Your table and its data stay in the warehouse, no longer updated. | | Drop the destination table | The integration is removed and the table is permanently deleted from your warehouse. | | Rename the destination table | The integration is removed and the table is renamed with a `_deleted_` suffix and timestamp, so the data stays recoverable. | Type the confirmation text, then choose **Delete**. ### When files are processed * **Email Webhook** — files are processed immediately, as soon as they're received. * **S3 and SFTP** — files are processed when a workflow that includes the integration runs. By default the integration is added to the standard daily workflow. * **Daasity Storage** — files are processed when the account's daily workflow runs. ### Troubleshooting and recommendations #### **Files** * **Nothing is being picked up** — run Test Expression to confirm your path and Date Pattern actually match the files on the Data Connection; check the File Processing Log to see whether the file was seen at all; widen the Lookback Window if files arrive later than expected. #### **Parsing** * **Every value lands in one column** — the Field Separator doesn't match the file. Check whether it's comma, pipe, or tab delimited. * **The header row is loaded as data** — Header Row Number is wrong. Set it to the line holding your headers, or 0 if the file has no header. * **Columns are loading as the wrong type** — Daasity detects column names but not data types, so set Format on each mapping yourself. As soon as any one mapping uses a type other than text, values are cast on load for the whole integration rather than rejected. #### **Data** * **Fewer rows in the destination than the source** — different rows are producing the same sync key and overwriting each other. Add columns until the combination is unique. * **Rows that should update are inserting as new** — a sync key column's value changes between runs. Rebuild the key from columns that don't change. Still stuck? Email with the integration name and, if the file came in by email webhook, the file you intended to send.