Data File Integrations (CSV, Delimited, XLS)

Load CSV, tab-separated, delimited, and Excel files from a connection you control into your Daasity warehouse on a recurring schedule.

Datafile Integrations load flat files — CSV, tab-separated, other delimited files, and Excel workbooks — from a connection you control into your Daasity warehouse on a recurring schedule. You set them up yourself, test that your files are being found before you save, and watch each run in a processing log. No developer and no support ticket required.

What a Datafile Integration is

A Datafile Integration picks up files from a location you already control — an SFTP server, an S3 bucket, Daasity Storage, or an address that receives files by email — parses them, and loads their rows into a table in your warehouse. Each run looks for files that match the path and date pattern you set, so once an integration is built it keeps loading new files on schedule without any further work from you.

Supported file types

File type	Notes
CSV	Comma-separated values.
Tab-separated	Columns separated by tabs.
Delimited	Any single-character delimiter — choose comma, pipe, or tab.
Excel	Reads a named worksheet from an .xlsx workbook. The delimiter option is hidden for Excel, because it doesn't apply.

How Datafile Integrations are organized

Every integration lives underneath a Data Connection. Understanding the relationship makes the rest of this article easier to follow.

• Data Connection (parent). The place your files live — SFTP, S3, Daasity Storage, or an email webhook — along with the credentials to reach it. You create it once and reuse it for as many integrations as you like.

• Datafile Integration (child). A single feed of files from that connection into one warehouse table. It holds the file path, the parsing settings, and the field mappings that turn each file into rows.

Naming note. The field once labeled Data File Source is now called Data Connection. You'll also see Data File Path/Name shortened to File Path, and Filename Date Pattern shortened to Date Pattern. The behavior is the same — only the labels changed.

Creating a Datafile Integration

Start from the Data Connection that holds your files and add a new integration for the file type you're loading. Fill in the sections top to bottom — each one is described in the field reference below — then save.

What you'll see when you save

Creating an integration happens right in front of you. A progress window opens and moves through three named steps in real time:

1. Creating Datafile Integration — your settings are saved. If anything on the form needs fixing, the form comes back with the errors marked and nothing is created.

2. Creating Warehouse Table — the destination schema and table are built in your warehouse while you wait, so you find out immediately if the table can't be created — not hours later on the first run.

3. Field Mappings Validation — your mappings are checked against the table that was just created. When everything passes, you land on your new integration, ready to run.

All or nothing. The whole create runs as a single transaction. If any step fails, the window stops on it and names the problem, and no half-built integration is left behind — just fix the error and save again. You no longer need to hunt down and delete a partly created integration after a failed save. The most common failure is two destination columns with the same name, which stops the run at step 3.

Field reference

Every field on the create and edit forms, with the help text you'll see on screen. The create and edit forms use the same wording, so a field means the same thing in both places.

General

Field	What it's for
Name	A label that identifies what this integration extracts. Free text — pick something you'll recognize later.
Data Connection	The connection where your files live (SFTP, S3, and so on). Pick an existing data connection or create a new one.
File Path	The path or prefix to your file(s) on the connection. Files matching this path are picked up on each run. You can include the token {{date}}, which expands to match the dated part of your filenames.
Date Pattern	If your filenames contain a date, choose its format so each run can read the date from the filename. Leave it as None when your filenames have no date. Options are listed below.
Header Row Number	Which line holds the column headers. Use 0 if the file has no header row — this one is easy to miss.
Field Delimiter	How columns are separated: Comma (,), Pipe (|), or Tab (\t). Hidden on Excel integrations, where it doesn't apply.
Sheet Name	Excel only — the worksheet to read from the workbook.

Date Pattern options

The dropdown lists the same set of date orderings across five separator styles. Pick the one that matches how the date appears in your filenames.

Separator	Example options
Hyphen	yyyy-mm-dd · dd-mm-yyyy · mm-dd-yyyy · yyyy-mm
Space	yyyy mm dd · dd mm yyyy · mm dd yyyy · yyyy mm
Underscore	yyyy_mm_dd · dd_mm_yyyy · mm_dd_yyyy · yyyy_mm
None	yyyymmdd · ddmmyyyy · mmddyyyy · yymmdd · yyyymm
Dot	yyyy.mm.dd · mm.dd.yyyy · dd.mm.yyyy

Check the order before you save. Similar patterns like yyyy-mm-dd and yyyy-dd-mm both exist, and choosing the wrong one quietly matches the wrong files instead of showing an error. Confirm your choice with Test Expression (see Finding your files) before saving.

Destination

Field	What it's for
Destination Schema Name	The schema in your warehouse where the table is created.
Destination Table Name	The name of the table in your warehouse.
Data Load Action	Whether each run replaces all data, or updates existing data. See Data loading.

Finding your files

Before you save, use Test Expression to confirm that your path and date pattern actually match files on the connection. It's the fastest way to catch a typo in a path or a mismatched date format — and the first thing to try whenever an integration "isn't picking up my file."

Detail	How it works
Where to find it	A Test Expression button in the General section, with the hint "Validate your path and date pattern against the datasource." It's hidden for email webhook and Daasity Storage connections, which have no browsable folder.
Enter a path first	The button stays disabled until you've entered a file path.
In the window	You'll see read-only copies of your path and Date Pattern, plus a Test Date field hinted "Used to narrow the file search around a specific date." Choose Run Test to search, or Cancel to close.
Results	Matching filenames stream back as they're found, up to 20 files, with a "showing N of M" line when more match. If nothing matches, you get a clean empty result rather than an error.

On SFTP, allow a moment. Testing against an SFTP connection can be a little slow, because each test opens a fresh connection to your server. That's expected — give it a few seconds to return.

Field mappings

Field mappings turn each column in your source file into a column in the destination table. Load a sample file to auto-detect the columns, then adjust the mapping table as needed.

What you can do in the mapping table:

• Include or exclude each column. Every row has a checkbox. Clear it to leave that source column out — excluded columns aren't created in the destination table at all. This is how you drop a source column you don't need.

• Set a date format per column. Choosing a date or timestamp type reveals a format picker, so a column like 03/04/2026 is read the way you intend rather than guessed.

• Catch duplicate names as you type. Entering a destination name that already exists in the mapping is flagged on the spot, instead of failing when you save.

• Sync-key choices stay in step. Renaming a destination column immediately refreshes the sync-key selectors, so they can never point at a name that no longer exists.

• Your choices survive an error. If a save fails, the table comes back with your selections intact instead of resetting.

Typed columns are formatted automatically. As soon as you give a mapping a type, Daasity formats that column when it loads — there's no separate box to tick. In practice this means a currency column such as $1,234.56 typed as numeric now loads correctly instead of rejecting the file.

Data loading

The Data Loading section controls how each run puts rows into the destination table, with an Advanced subsection for trickier source files.

Main controls

Control	What it does
Current sync key	A read-only display of the columns that currently form the key, with the note "Columns are always combined in alphabetical order."
Replication key type	Chooses how rows are identified from one run to the next.
Primary Key column	Use a single column as the key.
Destination columns to use	Use several columns together as the key. If there are no mappings yet, this reads "No field mappings available."
Do not remove duplicates	Keeps duplicate rows instead of de-duplicating on the key.
Lookback Window (days)	How far back each run re-reads. Leave it blank for the default of 2 days.

What is a sync key? A sync key is how Daasity tells whether an incoming row is new or an update to a row it already has. When rows arrive with a key that matches an existing row, that row is updated; when the key is new, a new row is added. Columns in the key are always combined in alphabetical order.

Lookback Window now works with any value. Setting Lookback Window (days) to any number is fully supported. If you were ever told to leave this field blank, that advice no longer applies — set it to whatever window your data needs.

Advanced parsing options

These options handle source files that don't parse cleanly on their own. The section is hinted "Advanced parsing options for tricky source files." Each option's on-screen description is quoted below.

Option	Description as shown	When to use it
Use positional header	Replace the original header with one built from field mappings in order.	Source files whose header row is unreliable or missing.
Sync key uses filename date	Prepends the date extracted from the filename to the sync key.	Daily snapshot files where the same row id recurs each day. Changing this re-keys existing rows — see Changing the sync key.
Remove escape chars	Strip backslash escape characters.	Files that use backslash escaping a normal parser mishandles.
Remove inner quotes	Remove quote characters that appear inside field values. Enable this when the source file contains unescaped inner quotes that break CSV parsing.	The classic "it's a "test"" → it's a test case.
Substitution dictionary	A JSON array of match/replace pairs applied to every field during processing. Leave blank to skip substitution.	Blanking out sentinel values, e.g. [{"match": "N/A", "replace": ""}]. Entries are validated when you save, so a malformed one is caught up front instead of failing on a run.

Changing the sync key

Changing an integration's sync key changes how rows are identified, which makes the rows already in the destination table unreachable under the new key. To prevent surprises, Daasity stops you and asks what to do before it saves.

Detail	What you'll see
Title	Sync Key Changed — Action Required
Question	What should happen to the destination table?
Options	Truncate — empty the table and reload it against the new key. Copy & truncate — copy the table to <table>_backup_YYYYMMDDHHMMSS first, then empty it.
To confirm	Type the table name, then choose Confirm & Save.

Recommended. Choose Copy & truncate unless you're certain you won't need the existing rows — it keeps a timestamped backup you can fall back to. If the warehouse step happens to fail, your integration settings are still saved and any backup already made is left in place, so you won't lose your edit.

The processing log

The Log answers the question that comes up most often: did my file arrive, and what happened to it? It's available on every datafile integration, for all four file types.

Detail	What's there
Columns	Filename · Processed At · Status · Rows Extracted · Workflow
Search & filter	Search by part of a filename, and filter by status. Shows 25 rows per page, newest first.
Empty state	"No processing history found."

"A run that saw nothing" vs. "no run at all." Runs that matched no files are recorded here too. A run in the log with zero rows extracted is a different situation from no run at all — the first means the schedule fired but found nothing matching your path, which usually points at the path or date pattern (start with Test Expression).

Files received by email

Some connections receive files as email attachments. Every email address set up on your account appears on the Email Webhooks page, so you can see at a glance which addresses are live and which still need finishing.

Column	What it shows
Email	The address files are sent to.
Status	Whether the address is set up and running (see below).
Destination	The integration the address feeds.
Last Received	When a file last arrived — shows Never if none has.

Statuses:

• Setup Incomplete — the address exists, but no integration is listening on it yet, so files sent to it won't load. Use the Finish Setup action to complete it.

• Active — wired to a live integration and receiving files.

• Inactive — the integration isn't currently running.

A webhook has just a name and a shop id, and you edit those on the integration's own form — there's no separate page to manage.

An address routes to one integration. A single email address can feed only one integration. When you're choosing an address for a new integration, addresses already in use are hidden from the list, so you can't accidentally point two integrations at the same inbox — which would leave one of them never receiving a file.

Deleting an integration

When you delete a datafile integration, the integration itself is always archived. What you choose only decides what happens to the warehouse table it was loading.

Option	What happens to the table
Keep	The table and its data are left untouched. This is the default.
Rename	The table name gets _deleted_YYYYMMDDHHMMSS appended, so the data stays queryable under the timestamped name.
Drop	The destination table is deleted.

You'll type the table name to confirm before the button enables. If you're unsure, Rename is the safe middle choice — the integration is gone, but the data is still there under its new name if you find you need it.

Troubleshooting

Common symptoms and where to look first. When in doubt, Test Expression and the processing log answer most "why isn't my file loading" questions between them.

Symptom	What to check
An integration isn't picking up my file.	Run Test Expression to confirm your path and date pattern match a real file. If nothing comes back, the path or the date pattern is the most likely cause.
My filename has a date, but the wrong files load.	Check the Date Pattern ordering — e.g. yyyy-mm-dd vs. yyyy-dd-mm. A mismatched order matches the wrong files silently.
My file has a space in its name.	Filenames with spaces are supported on S3 — you no longer need to rename source files as a workaround.
A file at the top level of my SFTP server won't load.	Files in the SFTP root directory now load correctly. If one still fails, confirm the path and re-run.
A currency or formatted number is rejecting the file.	Give that column a numeric type in the field mappings — typed columns are formatted on load, so values like $1,234.56 come in cleanly.
My run says it finished, but the table is empty.	Look in the processing log. A run with zero rows extracted means the schedule fired but matched no files — check the path and date pattern.
A file I emailed never appeared.	Open Files received by email and check the address's status. Setup Incomplete means nothing is listening yet — use Finish Setup. Confirm the address isn't one already used by another integration.
Credentials aren't being accepted.	Re-enter them without any trailing spaces or line breaks. Source filenames should also not contain line breaks.