Object Storage

Connect Gretel to object storage based services.

Gretel Workflows support connecting to the following object storage services

• Amazon S3

• Google Cloud Storage

• Azure Blob Storage

Reading Objects

Object storage source actions will incrementally crawl buckets searching for files that have changed between runs. Crawled files can then be configured as inputs to Gretel Models.

Each crawled object is passed as an input to the configured Gretel Model. Some models may need a certain amount of records, while other models might not work for datasets that are too large.

For best results, ensure objects contain the appropriate amount of records to successfully train and run the downstream model.

Writing Objects

Object storage destination actions can be configured to write the synthetic data outputs of a Gretel Model back to object storage.

Each object storage destination action can be configured to mirror the directory structure of the source bucket or can be configured to create new directory layouts.

Limitations

For a list of supported file types, please refer to Inputs and Outputs.