Narrative Snowflake Native App Data Flows

The Narrative Snowflake Native App integrates Snowflake with Narrative's platform, facilitating seamless data interaction and management directly within Snowflake's native environment. This article provides a deeper dive into the data flows between a customer's Snowflake account, the Narrative platform, and Connector Framework egress destinations.

Data Plane vs Control Plane Separation

Narrative Control Plane

• Managed by Narrative.
• Hosts metadata, user permissions, and configurations.
• Contains operational data like NQL query definitions, job statuses and logs.

Customer Data Planes

• Managed by the customer (e.g., Snowflake account).
• Hosts sensitive customer datasets.
• Contains the raw and processed data utilized in data queries and deliveries.
• Hosts the execution engine for all queries run by Narrative in the data plane (processing occurs in the customer's Snowflake account)
• All processing and storage costs for operating the Snowflake Native App occur in a customer's Snowflake account and will be billed by Snowflake as normal.

Data Flow

Data Integration

1. Connecting Snowflake to Narrative:
   • Users connect their Snowflake account to Narrative by installing the Narrative Snowflake Native App and providing their Narrative API Key.
   • This registers a new Data Plane in the Narrative platform that represents the customer's Snowflake account and region.
2. Registering Snowflake Tables:
   • Snowflake tables are registered as datasets within the Narrative platform.
   • Datasets contain metadata only, limited to the schema of a Snowflake table, additional metadata such as names and descriptions, and optionally sample data and dataset statistics.
   • Queries are executed within the Snowflake account, ensuring security and data privacy.

Data Management

• Queries created in the Narrative platform are translated to Snowflake SQL.
• These queries are executed in your Snowflake instance.
• Results are processed within Snowflake (generating new tables in your Snowflake account where necessary) without moving data externally.

Data Delivery

• Processed data can be delivered to various endpoints (e.g., The TradeDesk, Facebook) when a Connection is created between a dataset and an Egress Connector Destination.
• Data flows from Snowflake through the Narrative Connector Framework to the targeted destinations.
  • Note: In Alpha, this data makes a stop on Narrative's infrastructure before being cached and delivered to connector destination endpoints. In future versions of Narrative's Snowflake app, data will flow directly to connector destinations.

Snowflake Table Metadata to Control Plane

When integrating Snowflake with the Narrative platform, certain metadata may be shared with the Narrative Control Plane to optimize data management and operations. Here are the key types of metadata that may be shared:

1. Data Schema:
   • Field Names: Names of the columns in the table.
   • Field Types: Data types of each field (e.g., INTEGER, STRING).
2. Sample Data (Optional):
   • A sample of 1000 rows can be automatically shared.
   • This is optional and requires user opt-in by clicking "Send Sample Data" during or after configuration.
   • Users can alternatively call the Sample API directly with cleansed data or forgo sending sample data.
3. Dataset Statistics (Optional):
   • Column Count: Total number of columns in the table.
   • Unique Values: Count of unique values for each column.
   • Table Size: Size of the table in rows and bytes.
   • This requires user opt-in by clicking an appropriate button similar to sample data sharing.
4. Log Files (Optional):
   • Operational Metadata: Includes job statuses, execution times, and other operational information.

   Note on Logs

   
   Please note that logs generated and managed by the Narrative platform contain only metadata and operational information, such as job statuses and execution times. These logs do not include any rows of actual data from your datasets, ensuring the privacy and security of your data.