Daasity Knowledge Base
Daasity Knowledge Base
  • Start here
    • Daasity Overview
    • Important Concepts
      • Attribution Overview
      • What is a Daasity Valid Order?
      • Customer Householding
      • Timezones in Daasity
      • Ecommerce Glossary
      • Daasity Data Security Overview
    • Retail Analytics 101
      • Understanding the Basics: POS Data 101
      • How to use Syndicated Data
    • Product Changelog
    • Feature Requests
  • 🔌Extract
    • Extract your data with Daasity
    • All Integrations
      • Alchemer
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Algolia
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Amazon Advertising
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Where to Find Your Data
      • Amazon Seller Central
        • Validating Your Amazon Data in Daasity
          • Comparing Your Amazon Seller Central Orders to Daasity
        • Integration Setup
          • Configure Amazon Reports
          • Monitor Amazon Reports
        • Integration Specifications
          • Orders API
          • FBA Customer Concessions Reports
          • FBA Sales Reports
          • FBA Inventory Reports
          • FBA Removals Reports
          • Inventory Reports
          • Order Tracking Reports
          • Performance Reports
          • Returns Reports
          • Settlement Reports
          • Seller Retail Analytics Reports
        • Workflow Configuration Setup
          • Orders API
          • FBA Customer Concession Reports
          • FBA Inventory Reports
          • Performance Reports
          • FBA Sales Reports
          • FBA Removal Reports
          • Inventory Reports
          • Order Tracking Reports
          • Returns Reports
          • Seller Retail Analytics Reports
          • Settlement Reports
        • Transformation Configuration Setup
          • Orders API
          • FBA Customer Concessions Reports
          • FBA Inventory Reports
          • Performance Reports
          • FBA Sales Reports
          • FBA Removals Reports
          • Inventory Reports
          • Order Tracking Reports
          • Returns Reports
          • Seller Retail Analytics Reports
          • Settlement Reports
      • Amazon Vendor Central
        • Integration Setup
        • Integration Specifications
        • How to use
        • ASIN Mapping Configuration Setup
        • Workflow Configuration Setup
        • Transformation Configuration Setup
          • Vendor Sales & Inventory Report
          • Vendor Traffic Report
      • AppLovin
        • Integration Setup
      • Attentive Mobile
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Azure SQL Database
      • Back In Stock
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Big Commerce
        • Integration Setup
          • Connect Daasity From BigCommerce Marketplace
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • BigQuery Datasource
      • Bing Ads
        • Integration Setup
        • Integration Specifications
      • Circana
        • Integration Setup
        • Integration Specifications
        • Data Release Calendar
        • H-E-B Circana
      • Cordial
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Criteo
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • D​ata File (CSV, XLS)
      • Emplifi
        • Integration Specifications
      • Extensiv (Skubana)
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Facebook Ads
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Where to Find Your Data
      • Fairing V2
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Important Notes
      • Google Ads
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Important Notes
      • Google Analytics 4
        • API vs BigQuery
        • API Integration
          • Integration Setup
          • Integration Specifications
          • Workflow Configuration Setup
          • Transformation Configuration Instructions
          • Custom Reports
          • Validating GA4 Data in Daasity
          • Blending Universal Analytics & GA4 Data
        • BigQuery integration
          • Integration Setup
          • Integration Specifications
          • Workflow Configuration Setup
          • Transformation Configuration Instructions
          • Important Notes
          • Creating a Custom Report
      • Google Sheet
      • Gorgias
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Important Notes
      • Impact Radius (Beta)
        • Integration Specifications
      • Iterable
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Klaviyo Summary Statistics
        • Integration Setup
        • Integration Specifications
      • Klaviyo V2
        • Integration Setup
        • Integration Specifications
      • KnoCommerce
        • Integration Setup
        • Integration Specifications
      • Loop Returns
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Microsoft SQL Server Database
      • Mongo Database
      • MySQL Database
      • Narvar
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • NetSuite
      • Newstore
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Northbeam
        • Integration Setup
      • Okendo
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Ometria
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Order Desk
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Pepperjam
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Pinterest
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Postgres Database
      • Postscript (beta)
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Important Notes
      • Recharge
        • Integration Setup
        • Integration Specifications
        • Integration Specifications (2021-11 Beta)
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Recharge Data and Known Discrepancies
      • Retail Next
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Rockerbox
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Salesforce Commerce Cloud
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • SFTP
      • ShipBob
        • Integration Setup
        • Integration Specifications
      • Shipstation
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Shopify
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
        • Shopify B2B Data
      • SKIO
        • Integration Setup
        • Integration Specifications
      • Snapchat
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • SPINS
        • Integration Setup
        • Integration Specifications
        • How to Use
        • Data Release Calendar
      • Stay Ai
        • Integration Setup
        • Integration Specifications
      • Target Plus
        • Integration Setup
      • Thankful
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • TikTok
        • Integration Setup
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Walmart Marketplace
        • Integration Setup
        • Integration Specifications
      • Whole Foods Market
        • Integration Setup
        • Integration Specifications
        • How to use
      • Yotpo
        • Integration Setup
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
      • Zendesk
        • Integration Specifications
        • Workflow Configuration Setup
        • Transformation Configuration Setup
    • Brand Supplied Data (BSD)
      • BSD Overview
      • Monitoring Your BSD Progress
      • Revenue Plan
      • Marketing Budget
      • Marketing Spend
      • RFM Values
      • Shipping & Fulfillment Cost
      • SKU Hierarchy
      • SKU Mapping
      • SKU Cost
      • Discount Code Mapping
      • Other Order Sources
      • Channel Mapping
        • Overview & Setup
        • Filling out Channel Mapping
        • Missing from GA & Missing from BSD
      • Configuration
  • 📊VISUALIZE
    • Dashboards
      • Dashboard Overview
      • Standard Dashboards
        • Daasity 101 Overview Dashboards
          • Flashes
          • Shopify Sales Report
          • Account Health
        • Digital Product & Revenue
          • Orders and Revenue
          • Amazon Settlement Report
          • Product Repurchase Rate
          • Product
            • Product Dimensions
              • Why is a Product Name NULL?
          • Inventory
          • Operations
            • Fulfilment Status Options
        • Retention Marketing Dashboards
          • Notifications (Email/SMS)
          • Retention
          • Customer Performance (Email/SMS)
          • Site Funnels
        • Acquisition Marketing Dashboards
          • Attribution Deep Dive
            • Marketing Attribution
              • Attribution Overview
              • Survey-Based Attribution
              • Discount Code Attribution
              • First Click, Last Click, and Assisted Attribution
              • Custom Attribution
              • Vendor-Reported Attribution
              • CPO, CPA, and CAC
              • Missing from GA & Missing from BSD
          • LTV & RFM
            • RFM
              • RFM in the Daasity App
              • Understanding RFM
              • Time on Books (TOB)
          • Marketing
          • Vendor-Reported Marketing Performance
          • Site Analytics & Attribution
      • Integration Dashboards
        • Amazon All Orders Report
        • Amazon Seller Central Business Reports
        • Gorgias
        • Klaviyo Campaign & Flow Performance
        • Loop Returns
        • Okendo Reviews
        • ShipBob
        • Shipstation
        • Subscription
        • Walmart Marketplace Account Sales Report
        • Yotpo
    • Explores
      • Explore Overview
      • Order Line Revenue vs. Transactional Sales Report
      • How to Access An Explore
      • Standard Daasity Explores
        • Current Inventory Level Explore
        • Customer Details Explore
        • Daily Company Metrics Explore
        • Daily Marketing Plan To Actual Marketing Performance Explore
        • Daily Plan to Actual Explore
        • Lifetime Value Explore
        • Marketing Attribution Explore
        • Multi-Channel Transaction Explore
        • Order and Order Line Revenue Explore
        • Product Page Explore
        • Shopping Stage Explore
        • SKU Level Repurchase Rates Explore
        • Traffic Explore
        • Transactional Sales Report Explore
        • UA vs G4 Explore
      • Integration Explores
        • Algolia
        • Amazon Settlement Explore
        • Fairing Survey Responses Explore
        • Gorgias Tickets Explore
        • Loop Returns Explore
        • Okendo Reviews Explore
        • Subscribers and Subscription Monthly Churn Rates Explores (Recharge)
        • Shipstation Explore
        • Yotpo
        • Zen Desk
    • Looker Guides
      • Building visualizations
      • Working with date dimensions & filters
      • Schedule a Report or Dashboard
      • Exporting Data
    • Looker Guides (Developer)
      • Daasity Hub & Spoke Introduction
      • Hub & Spoke File Structure
      • Working with LookML Dashboards
      • Customizing Base Daasity Code
    • Visualize using other BI tools
  • 🔀Transform
    • Transforming Data with Daasity
    • Daasity Data Models
      • Daasity Data Model Overview
      • Unified Data Schemas
        • Unified Schema Introduction
        • Unified Order Schema (UOS)
        • Unified Notification Schema (UNS)
        • Unified Subscription Schema (USS)
        • Unified Marketing Schema (UMS)
        • Unified Traffic Schema (UTS)
        • Unified Wholesale Schema (UWS)
      • Data Mart Models
        • Data Mart Introduction
        • Data Reporting Platform (DRP)
        • Channel Attribution Data Mart
        • Marketing Data Mart
      • Transform Code
        • Transform Code Introduction
        • Transform Code Sequence - DRP
        • Initialization Code
        • Householding Code
        • Order Status Code
        • Order Channel and Marketing Code
        • Daily Plan Code
        • Order and Order Line Revenue
        • Plan to Actual Code
        • Customer Code
        • Subscription Code
        • Reviews
        • Life-Time Value Code
        • Returns
        • Product and Inventory Code
        • Email & SMS
        • Transform Code Sequence - Data Mart
        • Unified Order Schema (UOS) Code
        • Unified Marketing Schema (UMS) Code
        • Unified Notification Schema (UNS)
        • Channel Attribution Data Mart Code
        • Marketing Data Mart Code
      • Source to Final Destination
    • Code Repository
      • Code Repository Overview
      • Getting Access to Your Code Repository
      • Code Repository Branches
      • Custom SQL Code
      • Custom Script Manifest Files
    • Script Manifest (YAML) Files
  • 🎵Data Orchestration
    • Workflows
      • Getting Started with Workflows
      • Creating Workflows
      • Managing Workflows
      • Workflow Metrics
    • Notifications
      • Introduction to Notifications
      • Enabling Notification Emails
      • Notifications UI
      • Managing Notifications
      • Authorization List
  • Audiences
    • Audiences Overview
    • Pushing Audience Data to Targets
      • Targets Overview
      • Attentive
        • Events Target
        • Subscribers Target
        • Target Endpoint Documentation
      • Facebook Ads
        • Facebook Target
        • Target Endpoint Documentation
      • Google Ads
        • Google Ads Target
        • Target Endpoint Documentation
      • Klaviyo
        • Events Target
        • Profiles Target
        • Target Endpoint Documentation
      • Sailthru
        • Events Target
        • Users Target
        • Target Endpoint Documentation
      • Tiktok Ads
        • TikTok Target
        • Target Endpoint Documentation
    • Building Audiences
  • warehouse
    • BigQuery
    • Redshift
    • Snowflake
    • Test Data Warehouse
      • Test Data Warehouse Overview
      • Create a Snowflake Test Warehouse
      • Create a Redshift Test Warehouse
  • —
    • Go to Daasity App
    • Contact Support
    • ✍️Feature Requests
Powered by GitBook
On this page
  • Introduction
  • Example of running a simple transform script
  • Running pre-built Daasity SQL blocks
  • Using test warehouses

Was this helpful?

  1. Transform

Transforming Data with Daasity

Get an overview of how to transform the raw data in your data warehouse into analysis-ready models

PreviousVisualize using other BI toolsNextDaasity Data Models

Last updated 1 year ago

Was this helpful?

Introduction

You can use Daasity to transform your raw data into useful analytical models. You can do this by using custom transform code created by your team, or you can run a variety of pre-built SQL Code blocks maintained by Daasity.

When you transform data with Daasity, you will be using two different types of files:

  • SQL files that will contain your transformation code

  • Script manifest files, which are YAML files that tell Daasity which SQL scripts to run and in what order. .

You will deploy and manage all of these files within a custom code repository in GitHub, which we set up for you when you sign up for a Daasity account. .

Once you have some SQL files and at least one script manifest YAML file in your repository, you can execute the code using a workflow, which you create and manage within the Daasity app to orchestrate your transformations. You can specify in the app how often a workflow should run and whether any data extractions should run before running the transformation code. .

Example of running a simple transform script

We'll walk through the steps of testing your very first transformation script after .

Step 1: Create a custom SQL file in your custom repository

From the Code Repository page, navigate to your custom repository.

Add a file to your repository called demo_test.sql with the following contents, and commit the changes:

demo_test.sql
-- make sure the schema doesn't exist
DROP SCHEMA IF EXISTS daasity_demo;

-- create the schema
CREATE SCHEMA daasity_demo;

-- create a table
CREATE TABLE daasity_demo.demo
(
    id      VARCHAR
  , value   INTEGER
)
;

-- insert values
INSERT INTO daasity_demo.demo
VALUES
(1001,1),
(1002,2),
(1003,3)
;

-- select count and sum
SELECT COUNT(value), SUM(value)
FROM daasity_demo.demo
;

This code will create the table demo in schema daasity_demo in your data warehouse and insert some dummy data.

Step 2: Create a new script manifest file in your custom repository

Add a script manifest yml file to your repository called demo_test.yml with the following contents:

demo_test.yml
version:  '2.0.0'

sections:
  test:
    scripts:
      - "demo_test.sql"

You can read to understand the components of this code, but the gist is that it will run the demo_test.sql script you created in Step 1.

Step 3: Set up a workflow to run the script manifest file

Navigate to the Workflows section of the Daasity app and create a new workflow:

Name the workflow:

In the Data Transformation section, toggle on "Run transform scripts":

In the Data Transformation section, select the branch you used when creating the files in steps 1 & 2, and select the demo_test.yml script manifest file:

Click "Create" in the upper right corner of the screen

Step 4: Run the workflow & check your data

On the Workflows page, find the workflow you just created in the Configured Workflows section.

Then hover over the row, and click the "Run" button:

On the next screen, click the "Run" button in the upper right-hand corner:

Once the transformation part of the workflow has completed, check your data warehouse. You should now have a daasity_demo.demo table with 3 rows of data.

If the table did not get created, check your demo_test.sql and demo_test.yml files to make sure they match exactly the code from steps 1 and 2, and try running the workflow once more.

Running pre-built Daasity SQL blocks

Using test warehouses

This is an extremely simple example, but this is the basic process for all transformation that will occur within Daasity. When you're ready to set up more advanced workflows that incorporate data extractions or that run regularly at set intervals, read our and articles.

The above example shows how you can run your own custom SQL code. But if you're using our Transform code feature, you also have the option of running that is maintained by Daasity.

This code lives in the Daasity shared repository, which you to when you opt-in to our Transform code license. This shared repository gives you access to all of the code you would need to model your data using our .

You can run the shared code by . Doing so will always run the most up-to-date version of the Daasity code. Alternatively, you could use the Daasity code as a starting point and customize it to your business needs within your own custom code repository.

Our test warehouse feature makes it easy for you test your transformation scripts without putting your production data in jeopardy. From within the Daasity app, you can make a clone of your production warehouse within minutes that you can use to test your transform scripts. .

🔀
Learn more about script manifest files here
Read more about code repositories here
You can get all the details on workflows here
getting access to your code repository
our script manifest file article
Getting Started with Workflows
Creating Workflows
pre-built transform code
get access
Daasity Data Models
Learn more about test warehouses here
referencing the shared scripts from your script manifest files