Knowledge Base

Setting Up the AWS S3 Connector App

Prerequisites

• Ensure you have an AWS account. Create one here if needed.

Installing the S3 Connector

1. Access the S3 Connector App via the apps tab in Narrative's platform.
2. Click the install button on the app's tile. Contact support@narrative.io for installation issues.

Configuring Bucket Profiles

1. Launch the App: Open the S3 Connector App and select "Connector Settings".
2. Create a New Profile: Click "+ New Profile", then enter your profile name, description, and AWS S3 bucket name.
   • Profile Name: Use a descriptive name, such as "North American Location Data".
   • Description: Detail the use case for easier internal recognition.
   • Bucket Name: The exact name of your S3 bucket (e.g., "narrative-testing-bucket").

Granting Narrative Access

1. Bucket Policy: Add the provided policy to your S3 bucket via the AWS console under the Permissions tab.
2. Bucket Tag: Add the provided Tag Key and Value to your bucket for verification.

Finalizing Setup

1. Test Connection: Ensure the setup is correct by testing the connection in the S3 Connector App.
2. File Configuration: Choose your preferred file type (CSV, TSV, JSON, or Parquet) and configure as necessary for CSV and TSV.

Click "Save And Finish" to complete the setup.

Using Your S3 Destination During Checkout

1. Select S3 Connector: In Data Studio's metadata section or the connections section of a dataset, select the S3 connector and choose your profile.
2. Multiple Profiles: To send data to multiple buckets, select "Add An Additional Profile" and configure each as needed.
3. Advanced Settings: The S3 Connector offers advanced settings that can be configured to provide more control over how data is processed and delivered to your AWS S3 bucket.* Deliver All Data: When enabled, this option allows the delivery of data currently present in the dataset. If disabled, only new data written to the dataset after the connection has been configured will be delivered.
   • Remove Metadata from Output Files: When set to true, the remove_metadata option instructs the S3 Connector to remove all fields beginning with _nio from the delivered data such as source and license information. This is particularly useful for users looking to streamline their datasets by excluding metadata that may not be relevant to their analysis or storage needs.
   • Use Delivery Date in Output Path: If enabled, new files will be prefixed with the current date.
   • Include Success File: If enabled, an empty file with the name '_SUCCESS' will be written into the output folder to indicate that all files have been successfully written and the delivery is complete.* The _SUCCESS file is created once per delivery to indicate successful completion. If all deliveries are directed to the same folder (for example, if you use the date folder), only one _SUCCESS file will exist, and it will be overwritten with each new delivery. This means that if you have multiple deliveries in one day, the interaction with the "use delivery date" option can cause the _SUCCESS file to be overwritten multiple times within that day.

Confirm the settings and complete the checkout process to start receiving data in your specified S3 bucket. This advanced functionality provides users with greater flexibility and control over their data deliveries, allowing for customization to meet specific needs or preferences.

API Configuration

Configuration Options

Below is an example of advanced settings available through the API:

curl --location --request POST 'https://app.narrative.io/openapi/connections' \
--header 'Authorization: Bearer {{Bearer-Token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "dataset_id": 12345,
    "profile_id": "XXXXXXX-XXXXX-XXXXXX",
    "quick_settings": {
        "bucket_prefix": "/my-prefix",
        "historical_data_enabled": true,
        "remove_metadata": false,
        "use_delivery_date": true,
        "include_success_file": true
    }
}'

For further assistance or to enable these advanced settings, please contact support@narrative.io.