Getting Started with Workflows
This page provides an introduction to workflows, which are core to the Daasity platform and enable you to build an integrated extraction and transformation process
Last updated
This page provides an introduction to workflows, which are core to the Daasity platform and enable you to build an integrated extraction and transformation process
Last updated
Workflows orchestrate all of the data extractions and transformation that occur within Daasity.
Each individual workflow can kick off a set of data extractions and then, once the data has been loaded, run any transformation code tied to the workflow.
Workflows are a powerful tool because you can tie transformation code to an extractor so that your SQL transformations kick off as soon as new data is ready. You can also configure them to kick off at different intervals and can chain workflows so that a subsequent workflow will start as soon as the chained workflow has completed.
We set up a daily incremental workflow for you when you set up Daasity, but you can also set up custom workflows that can run as often as once per hour.
You can get to the Workflows screen by navigating to the Workflows section on the left-nav of the Daasity app:
When you set up Daasity, a daily incremental workflow will automatically be created and will start running every night at midnight (according to the timezone you specify in the Daasity App settings). Any new integration that you set up will be added to the daily incremental workflow automatically.
When the daily incremental workflow starts, it will first extract data from all integrations included in the workflow and then load the data into your data warehouse.
Then, the workflow will use a script manifest file called daily_incremental.yml to execute transformation code. The purpose of daily_incremental.yml and other script manifest files is to list out a set of SQL scripts that will run in order when the transformation part of the workflow begins.
When it's initially set up, your daily_incremental.yml file will not contain any scripts, and no transform code will run. If you want to set up some transformation code you need to:
Get access to your code repository, which is where you'll find your daily_incremental.yml file. This will also give you access to Daasity's shared script repo if you are using our transformation code license.
Add SQL scripts to your daily_incremental.yml file and deploy the changes. Our help article on script manifest files will have more details on how to properly set up and modify this file.
You're not limited to using only the daily incremental workflow. You can set up as many additional workflows as you need, and you can set them up to run as frequently as once per hour.
You can also set custom workflows up to be extractor only or transformation only.
