> For the complete documentation index, see [llms.txt](https://help.daasity.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.daasity.com/core-concepts/data-integrations/custom-data-inputs/data-file-integrations-csv-delimited-xls.md). # Data File Integrations (CSV, Delimited, XLS) Load CSV, tab-separated, delimited, and Excel files from a connection you control into your Daasity warehouse on a recurring schedule. Datafile Integrations load flat files — CSV, tab-separated, other delimited files, and Excel workbooks — from a connection you control into your Daasity warehouse on a recurring schedule. You set them up yourself, test that your files are being found before you save, and watch each run in a processing log. No developer and no support ticket required. ### What a Datafile Integration is A **Datafile Integration** picks up files from a location you already control — an SFTP server, an S3 bucket, Daasity Storage, or an address that receives files by email — parses them, and loads their rows into a table in your warehouse. Each run looks for files that match the path and date pattern you set, so once an integration is built it keeps loading new files on schedule without any further work from you. #### Supported file types | File type | Notes | | ----------------- | --------------------------------------------------------------------------------------------------------------------- | | **CSV** | Comma-separated values. | | **Tab-separated** | Columns separated by tabs. | | **Delimited** | Any single-character delimiter — choose comma, pipe, or tab. | | **Excel** | Reads a named worksheet from an `.xlsx` workbook. The delimiter option is hidden for Excel, because it doesn't apply. | ### How Datafile Integrations are organized Every integration lives underneath a **Data Connection**. Understanding the relationship makes the rest of this article easier to follow. * **Data Connection (parent).** The place your files live — SFTP, S3, Daasity Storage, or an email webhook — along with the credentials to reach it. You create it once and reuse it for as many integrations as you like. * **Datafile Integration (child).** A single feed of files from that connection into one warehouse table. It holds the file path, the parsing settings, and the field mappings that turn each file into rows. {% hint style="info" %} **Naming note.** The field once labeled *Data File Source* is now called **Data Connection**. You'll also see *Data File Path/Name* shortened to **File Path**, and *Filename Date Pattern* shortened to **Date Pattern**. The behavior is the same — only the labels changed. {% endhint %} ### Creating a Datafile Integration Start from the **Data Connection** that holds your files and add a new integration for the file type you're loading. Fill in the sections top to bottom — each one is described in the field reference below — then save. #### What you'll see when you save Creating an integration happens right in front of you. A progress window opens and moves through three named steps in real time: 1. **Creating Datafile Integration** — your settings are saved. If anything on the form needs fixing, the form comes back with the errors marked and nothing is created. 2. **Creating Warehouse Table** — the destination schema and table are built in your warehouse while you wait, so you find out immediately if the table can't be created — not hours later on the first run. 3. **Field Mappings Validation** — your mappings are checked against the table that was just created. When everything passes, you land on your new integration, ready to run. {% embed url="" %} {% hint style="success" %} **All or nothing.** The whole create runs as a single transaction. If any step fails, the window stops on it and names the problem, and **no half-built integration is left behind** — just fix the error and save again. You no longer need to hunt down and delete a partly created integration after a failed save. The most common failure is two destination columns with the same name, which stops the run at step 3. {% endhint %} ### Field reference Every field on the create and edit forms, with the help text you'll see on screen. The create and edit forms use the same wording, so a field means the same thing in both places. #### General | Field | What it's for | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | A label that identifies what this integration extracts. Free text — pick something you'll recognize later. | | **Data Connection** | The connection where your files live (SFTP, S3, and so on). Pick an existing data connection or create a new one. | | **File Path** | The path or prefix to your file(s) on the connection. Files matching this path are picked up on each run. You can include the token `{{date}}`, which expands to match the dated part of your filenames. | | **Date Pattern** | If your filenames contain a date, choose its format so each run can read the date from the filename. Leave it as **None** when your filenames have no date. Options are listed below. | | **Header Row Number** | Which line holds the column headers. **Use `0` if the file has no header row** — this one is easy to miss. | | **Field Delimiter** | How columns are separated: **Comma (,)**, **Pipe (\|)**, or **Tab (\t)**. Hidden on Excel integrations, where it doesn't apply. | | **Sheet Name** | Excel only — the worksheet to read from the workbook. | #### Date Pattern options The dropdown lists the same set of date orderings across five separator styles. Pick the one that matches how the date appears in your filenames. | Separator | Example options | | -------------- | ---------------------------------------------------------- | | **Hyphen** | `yyyy-mm-dd` · `dd-mm-yyyy` · `mm-dd-yyyy` · `yyyy-mm` | | **Space** | `yyyy mm dd` · `dd mm yyyy` · `mm dd yyyy` · `yyyy mm` | | **Underscore** | `yyyy_mm_dd` · `dd_mm_yyyy` · `mm_dd_yyyy` · `yyyy_mm` | | **None** | `yyyymmdd` · `ddmmyyyy` · `mmddyyyy` · `yymmdd` · `yyyymm` | | **Dot** | `yyyy.mm.dd` · `mm.dd.yyyy` · `dd.mm.yyyy` | {% hint style="warning" %} **Check the order before you save.** Similar patterns like `yyyy-mm-dd` and `yyyy-dd-mm` both exist, and choosing the wrong one quietly matches the wrong files instead of showing an error. Confirm your choice with **Test Expression** (see Finding your files) before saving. {% endhint %} #### Destination | Field | What it's for | | --------------------------- | ------------------------------------------------------------------------------- | | **Destination Schema Name** | The schema in your warehouse where the table is created. | | **Destination Table Name** | The name of the table in your warehouse. | | **Data Load Action** | Whether each run replaces all data, or updates existing data. See Data loading. | ### Finding your files Before you save, use **Test Expression** to confirm that your path and date pattern actually match files on the connection. It's the fastest way to catch a typo in a path or a mismatched date format — and the first thing to try whenever an integration "isn't picking up my file." | Detail | How it works | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Where to find it** | A **Test Expression** button in the **General** section, with the hint *"Validate your path and date pattern against the datasource."* It's hidden for email webhook and Daasity Storage connections, which have no browsable folder. | | **Enter a path first** | The button stays disabled until you've entered a file path. | | **In the window** | You'll see read-only copies of your path and **Date Pattern**, plus a **Test Date** field hinted *"Used to narrow the file search around a specific date."* Choose **Run Test** to search, or **Cancel** to close. | | **Results** | Matching filenames stream back as they're found, up to **20 files**, with a "showing N of M" line when more match. If nothing matches, you get a clean empty result rather than an error. | {% hint style="info" %} **On SFTP, allow a moment.** Testing against an SFTP connection can be a little slow, because each test opens a fresh connection to your server. That's expected — give it a few seconds to return. {% endhint %} ### Field mappings Field mappings turn each column in your source file into a column in the destination table. Load a sample file to auto-detect the columns, then adjust the mapping table as needed. What you can do in the mapping table: * **Include or exclude each column.** Every row has a checkbox. Clear it to leave that source column out — excluded columns aren't created in the destination table at all. This is how you drop a source column you don't need. * **Set a date format per column.** Choosing a date or timestamp type reveals a format picker, so a column like `03/04/2026` is read the way you intend rather than guessed. * **Catch duplicate names as you type.** Entering a destination name that already exists in the mapping is flagged on the spot, instead of failing when you save. * **Sync-key choices stay in step.** Renaming a destination column immediately refreshes the sync-key selectors, so they can never point at a name that no longer exists. * **Your choices survive an error.** If a save fails, the table comes back with your selections intact instead of resetting. {% hint style="success" %} **Typed columns are formatted automatically.** As soon as you give a mapping a type, Daasity formats that column when it loads — there's no separate box to tick. In practice this means a currency column such as `$1,234.56` typed as numeric now loads correctly instead of rejecting the file. {% endhint %} ### Data loading The **Data Loading** section controls how each run puts rows into the destination table, with an **Advanced** subsection for trickier source files. #### Main controls | Control | What it does | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | **Current sync key** | A read-only display of the columns that currently form the key, with the note *"Columns are always combined in alphabetical order."* | | **Replication key type** | Chooses how rows are identified from one run to the next. | | **Primary Key column** | Use a single column as the key. | | **Destination columns to use** | Use several columns together as the key. If there are no mappings yet, this reads *"No field mappings available."* | | **Do not remove duplicates** | Keeps duplicate rows instead of de-duplicating on the key. | | **Lookback Window (days)** | How far back each run re-reads. Leave it blank for the default of 2 days. | {% hint style="info" %} **What is a sync key?** A **sync key** is how Daasity tells whether an incoming row is new or an update to a row it already has. When rows arrive with a key that matches an existing row, that row is updated; when the key is new, a new row is added. Columns in the key are always combined in alphabetical order. {% endhint %} {% hint style="success" %} **Lookback Window now works with any value.** Setting **Lookback Window (days)** to any number is fully supported. If you were ever told to leave this field blank, that advice no longer applies — set it to whatever window your data needs. {% endhint %} #### Advanced parsing options These options handle source files that don't parse cleanly on their own. The section is hinted *"Advanced parsing options for tricky source files."* Each option's on-screen description is quoted below. | Option | Description as shown | When to use it | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Use positional header** | *Replace the original header with one built from field mappings in order.* | Source files whose header row is unreliable or missing. | | **Sync key uses filename date** | *Prepends the date extracted from the filename to the sync key.* | Daily snapshot files where the same row id recurs each day. Changing this re-keys existing rows — see Changing the sync key. | | **Remove escape chars** | *Strip backslash escape characters.* | Files that use backslash escaping a normal parser mishandles. | | **Remove inner quotes** | *Remove quote characters that appear inside field values. Enable this when the source file contains unescaped inner quotes that break CSV parsing.* | The classic `"it's a "test""` → `it's a test` case. | | **Substitution dictionary** | *A JSON array of match/replace pairs applied to every field during processing. Leave blank to skip substitution.* | Blanking out sentinel values, e.g. `[{"match": "N/A", "replace": ""}]`. Entries are validated when you save, so a malformed one is caught up front instead of failing on a run. | ### Changing the sync key Changing an integration's sync key changes how rows are identified, which makes the rows already in the destination table unreachable under the new key. To prevent surprises, Daasity stops you and asks what to do before it saves. | Detail | What you'll see | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Title** | **Sync Key Changed — Action Required** | | **Question** | **What should happen to the destination table?** | | **Options** | **Truncate** — empty the table and reload it against the new key. **Copy & truncate** — copy the table to `_backup_YYYYMMDDHHMMSS` first, then empty it. | | **To confirm** | Type the table name, then choose **Confirm & Save**. | {% hint style="info" %} **Recommended.** Choose **Copy & truncate** unless you're certain you won't need the existing rows — it keeps a timestamped backup you can fall back to. If the warehouse step happens to fail, your integration settings are still saved and any backup already made is left in place, so you won't lose your edit. {% endhint %} ### The processing log The **Log** answers the question that comes up most often: *did my file arrive, and what happened to it?* It's available on every datafile integration, for all four file types. | Detail | What's there | | ------------------- | ----------------------------------------------------------------------------------------- | | **Columns** | **Filename** · **Processed At** · **Status** · **Rows Extracted** · **Workflow** | | **Search & filter** | Search by part of a filename, and filter by status. Shows 25 rows per page, newest first. | | **Empty state** | *"No processing history found."* | {% hint style="info" %} **"A run that saw nothing" vs. "no run at all."** Runs that matched **no files** are recorded here too. A run in the log with zero rows extracted is a different situation from no run at all — the first means the schedule fired but found nothing matching your path, which usually points at the path or date pattern (start with Test Expression). {% endhint %} ### Files received by email Some connections receive files as email attachments. Every email address set up on your account appears on the **Email Webhooks** page, so you can see at a glance which addresses are live and which still need finishing. | Column | What it shows | | ----------------- | ------------------------------------------------------- | | **Email** | The address files are sent to. | | **Status** | Whether the address is set up and running (see below). | | **Destination** | The integration the address feeds. | | **Last Received** | When a file last arrived — shows **Never** if none has. | Statuses: * **Setup Incomplete** — the address exists, but no integration is listening on it yet, so files sent to it won't load. Use the **Finish Setup** action to complete it. * **Active** — wired to a live integration and receiving files. * **Inactive** — the integration isn't currently running. A webhook has just a name and a shop id, and you edit those on the integration's own form — there's no separate page to manage. {% hint style="warning" %} **An address routes to one integration.** A single email address can feed only one integration. When you're choosing an address for a new integration, addresses already in use are hidden from the list, so you can't accidentally point two integrations at the same inbox — which would leave one of them never receiving a file. {% endhint %} ### Deleting an integration When you delete a datafile integration, the integration itself is always archived. What you choose only decides what happens to the **warehouse table** it was loading. | Option | What happens to the table | | ---------- | --------------------------------------------------------------------------------------------------------------- | | **Keep** | The table and its data are left untouched. This is the default. | | **Rename** | The table name gets `_deleted_YYYYMMDDHHMMSS` appended, so the data stays queryable under the timestamped name. | | **Drop** | The destination table is deleted. | You'll type the table name to confirm before the button enables. If you're unsure, **Rename** is the safe middle choice — the integration is gone, but the data is still there under its new name if you find you need it. ### Troubleshooting Common symptoms and where to look first. When in doubt, **Test Expression** and the processing log answer most "why isn't my file loading" questions between them. | Symptom | What to check | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | An integration isn't picking up my file. | Run Test Expression to confirm your path and date pattern match a real file. If nothing comes back, the path or the date pattern is the most likely cause. | | My filename has a date, but the wrong files load. | Check the **Date Pattern** ordering — e.g. `yyyy-mm-dd` vs. `yyyy-dd-mm`. A mismatched order matches the wrong files silently. | | My file has a space in its name. | Filenames with spaces are supported on S3 — you no longer need to rename source files as a workaround. | | A file at the top level of my SFTP server won't load. | Files in the SFTP root directory now load correctly. If one still fails, confirm the path and re-run. | | A currency or formatted number is rejecting the file. | Give that column a numeric type in the field mappings — typed columns are formatted on load, so values like `$1,234.56` come in cleanly. | | My run says it finished, but the table is empty. | Look in the processing log. A run with zero rows extracted means the schedule fired but matched no files — check the path and date pattern. | | A file I emailed never appeared. | Open Files received by email and check the address's status. **Setup Incomplete** means nothing is listening yet — use **Finish Setup**. Confirm the address isn't one already used by another integration. | | Credentials aren't being accepted. | Re-enter them without any trailing spaces or line breaks. Source filenames should also not contain line breaks. |