Workflows & Scheduling

Workflows allow you to schedule and control when data is extracted and transformed. Here’s what you need to know about Daasity workflows:

• What is a Workflow? It’s essentially a scheduled pipeline. A workflow is a configured sequence that first runs extractions for a set of sources, then runs the transformation scripts once the data is loaded. Instead of manually running dozens of jobs, a workflow automates the entire daily (or hourly) update process.

• Default Daily Workflow: When you first set up Daasity, the system creates a Daily Incremental Workflow for you. This daily workflow is scheduled to run once per day (commonly around midnight in your chosen timezone). It will automatically include all your data sources. Each night, it will pull the latest data from each integration and then update all the transformed tables. This keeps your unified data model and metrics up to date every day without manual intervention.

• Custom Workflows: You can create additional workflows for different purposes. For example, you might have a separate workflow to handle a large historical reload, or one that runs more frequently for a specific data source. Workflows can be scheduled as often as once per hour if needed, or just run on-demand. They are flexible: you choose which integrations and which transform scripts are tied to each workflow.

• Chaining and Dependencies: Daasity allows workflows to be chained. This means you can set one workflow to kick off another when it finishes. For instance, you might separate your workflows by domain (e.g., run an eCommerce data workflow, then a marketing data workflow afterwards). Chaining ensures that, say, your marketing metrics workflow only runs after all the purchase data from the eCommerce workflow is updated.

• Monitoring & Metrics: The Daasity app provides a Workflows dashboard where you can monitor progress and see metrics about each run (like duration, records processed, etc.). If a workflow fails at any step (maybe an API was down or a SQL script error), it will stop and flag an error. This is where Notifications come in to alert you. You can manually re-run workflows or individual steps if needed (for example, rerun just one integration’s extract if it failed).