Workflow Configuration Setup

This page provides information on how the HEB Circana integration can be configured as part of a workflow to extract data.

Extraction Replication Window

The H-E-B Circana integration processes data whenever the extractor runs in a workflow, after CSV files are uploaded. The workflow runs nightly by default. Each file upload updates existing records for matching geography, time period, and product combinations, and adds new records for any new combinations.

Important: Both integrations (HEB Circana Data and HEB Circana Products) must have files uploaded and tables created. The transformation pipeline will fail if both tables (retail.heb_circana_data and retail.heb_circana_products) do not exist.

Extraction Frequency

The daily extractor automatically fetches files from S3 or manual upload locations. The extractor will process files from up to 3 days ago.

Important: The extractor does not detect unprocessed files. Instead, it processes files based on the dates contained within the files themselves. Files with dates outside the expected range may not be loaded, even if the files are present in the upload location. This date-based processing approach is standard for all data file extractors.

File Naming Requirements:

• Files must follow the naming pattern: heb_circana_data*.csv and heb_circana_products*.csv

• Files should include a date in the filename using the format YYYYMMDD (e.g., heb_circana_data20240101.csv)

• Important: Files that do not match the expected naming pattern will be skipped by the extractor

Upload Methods:

• Manual upload through the UI

• Automated S3 delivery

Recommendations:

• Upload files weekly or bi-weekly, aligned with your HEB Circana data delivery schedule

• Upload files as soon as they are received from Circana to minimize data latency

• Ensure files are named correctly to avoid being skipped by the extractor

Processing Time

Files are processed automatically when the daily workflow containing the extractor runs. Data typically appears in reports within minutes to hours after the workflow run (depending on file size).

Note: The transformation will process any available data from both tables once they exist. However, the transformation will fail if both tables (retail.heb_circana_data and retail.heb_circana_products) do not exist.

Data Latency

Expected Timeline:

• File Delivery: Typically 1-2 weeks after period end (depends on HEB Circana delivery schedule)

• Upload & Processing: Files are automatically picked up by the daily extractor, which processes files from up to 3 days ago

• Total Latency: Typically 1-3 weeks from period end to reporting availability in Looker