Update your Data Product from an Endpoint

Once a repeating data product has been created it must be refreshed with updated data content. This is a two step process that is undertaken off-platform - you must first prepare your data and then trigger the publish.

Prepare your data

Upload the refreshed data content to a new subdirectory underneath the base location of the product in the object store source (e.g. AWS S3, GCP Cloud Storage, Azure Blob storage).

Note: This folder can have any name other than ‘singleload' which is a protected convention for the first data load when creating a product in the UI.

Warning: The data loaded into the new sub-folder must follow the same structure as the original data load used to create the product. This includes the same set of folders representing the tables in the product, the same data format (e.g. Parquet) and the same schema for each table.

Trigger the publish

Once refreshed data content has been loaded to the endpoint location in step 1, a product update is initiated by copying a Transfer Notification File (TNF) into the /tnfs/ subdirectory of the product endpoint location.

A TNF is a JSON text file containing the name of the newly created unique subdirectory containing the updated content - an example is shown below:

{"DataFolder":"newly_created_folder"}

CODE

where newly_created_folder is replaced by the newly created folder in the endpoint location (e.g. second_load).

The TNF file:

Must have the extension .tnf
Can be named using whichever convention you wish.
Must have the structure shown above.

The Data Exchange monitors the /tnfs/ subdirectories of repeating data products on a regular basis (parameterisable frequency with a default of 5 minutes), and when detected, initiates a load of the refreshed data from the sub-directory specified in the “DataFolder” JSON field.

When the data load process is complete the provided tnf will be moved to the /tnfs_complete/ subdirectory of the data product endpoint location with an additional field containing the status of the data ingest job, e.g.:

{"DataFolder":"newly_created_folder"
 "status":"success"}

CODE

Cloud storage event triggers can be set-up on the /tnfs_complete/ subdirectory of data products to facilitate programmatic monitoring of data load job execution status. Additional JSON fields can be added to the provided TNF file to facilitate such processes as required as these will be persisted to the output file.

You can view the status of your data product updates at any time.

References and FAQs

Transfer Notification File (TNF)