Multi-table Transformations

Multi-table functionality is currently in review for adoption and strategic fit. Please proceed with caution as it may be deprecated in the future.

Introduction

The Gretel Transform model can be applied to multiple related tables in a database at once, providing structured transformations without losing referential integrity across tables.

This functionality is executed through Gretel Workflows.

High Level Flow

1. Use a native connector to extract data from your source.

2. Train and run models via the gretel_tabular action.

3. Optionally, write output data to a destination sink.

4. Optionally, write output reports to an object store of your choice.

Gretel Tabular

The gretel_tabular action can be used to train and generate records from Gretel Models. It helps maintain referential integrity between related tables. gretel_tabular also allows specifying different model configs for different tables. This functionality is currently available only via the SDK. Read about Gretel Tabular.

Example Workflow

name: my-{connection_type}-workflow

actions:
  - name: {connection_type}-read
    type: {connection_type}_source
    connection: {input_connection_uid}

  - name: model-train-run
    type: gretel_tabular
    input: {connection_type}-read
    config:
      project_id: {project.project_guid}
      train:
        model_config:
          from: "transform/default"
        dataset: "{{{connection_type}-read.outputs.dataset}}"

Notebook Resources

gretel-blueprints/transform_relational_database.ipynb at develop · gretelai/gretel-blueprintsGitHub

Sample source connection

The example notebooks above use a special connection, sample_mysql_telecom, which connects to a demo telecommunications database:

ERD for mock telecommunications database