my-connector-sourcebucket prefixed with
sandboxwill be transformed so any PII is removed. After the S3 object has been de-identified, the object will be written into the
my-connector-destinationbucket and prefixed with
bucket- The name of the source bucket to ingest data from.
path_prefix- Objects matching this prefix will be processed through the connector. If the object does not match the prefix, that object will be skipped.
glob_filter- Filters for objects matching a specific glob filter. This is useful for filtering objects by file type. If an object does not match the filter, it will be omitted from processing. Glob filters follow standard unix style pathname pattern expansion.
trigger- The S3 connector is built to continuously poll for new objects arriving in a bucket. A trigger config defines where to poll new objects. Currently, only SQS triggers are supported.
type: sqs- This configures the connector to continuously poll a SQS queue for new S3 change events.
endpoint- Specifies the SQS endpoint to poll for new events.
bucket- The destination bucket to write objects back to.
path_prefix- Rewrites the source object path to the specified path prefix.