Connectors

Integrate Gretel with your existing data services using Workflows.

With Gretel Workflows, you can train and run one or more Gretel Models by connecting directly to your data sources and destinations. We support the following integrations for data inputs and outputs.

Supported Connectors

Object Storage

• Amazon S3

• Google Cloud Storage

• Azure Blob Storage

Relational Databases

• MySQL

• PostgreSQL

• Microsoft SQL Server

• Oracle Database

Data Warehouses

• Snowflake

• Google BigQuery

• Databricks

Managing Connections

Connections are used to authenticate Workflow Actions. Each action is compatible with a specific type of connection. For example, the s3_source action requires a s3 connection.

Permissions and Connection Sharing

When creating a connection, you must select a project for the connection to reside in. The connection will inherit all the project permissions and user memberships.

Project Permission	Connection Permissions
Read	Users can list and view connection metadata.
Write	Users can access connections in a Workflow.
Administrator	Users can create, edit and delete connections.
Owner	Full control.

Connections can be reused across Workflows, actions, and projects.

Creating Connections

If you're creating Connections in a Hybrid environment, follow along here: Connectors in Hybrid

CLI

First, create a file on your local computer containing the connection credentials. This file should also include type, name, config and credentials. The config and credentials field should contain fields that are specific to the connection being created.

Below is an example S3 connection

{
  "type": "s3",
  "name": "my-s3-connection",
  "config":{
    "access_key_id": "..."
  },
  "credentials": {
    "secret_access_key": "..."
  }
}

Now that you've created the credentials file, use the CLI to create the connection

gretel connections create --project [project id] --from-file [credential_file.json]

Console

• Navigate to the Connections page using the menu item in the left sidebar.

• Click the New Connection button in the top right corner.

• Select the Connector Type.

• Select the project where the connection will be stored.

• Next, fill in your credentials and select Add Connection. The example below shows the flow for creating an Amazon S3 connection. All connections follow the same pattern.

• The connection can now be used in a workflow.

Create a connection in the Gretel Console

• Data sources also can be configured during a blueprint flow. Go to the Dashboard or Blueprints page and select a use case, for example "Generate synthetic data". This will start a guided flow to help you create a workflow.

• Select the option to connect to an external data source, and choose the connection you created above, or create a new one.

Create a new connection during a workflow

SDK

from gretel_client import create_or_get_unique_project
from gretel_client.config import get_session_config
from gretel_client.rest_v1.api.connections_api import ConnectionsApi
from gretel_client.rest_v1.models import (
    CreateConnectionRequest,
    UpdateConnectionRequest,
)

session = get_session_config()
connection_api = session.get_v1_api(ConnectionsApi)

project = create_or_get_unique_project(name="workflow-testing")

connection = connection_api.create_connection(
    CreateConnectionRequest(
        name="my-s3-conn",
        project_id=project.project_guid,
        type="s3",
        # note: best practice is to read in credentials from a file
        # or secret instead of directly embedding sensitive values
        # in python code.
        config={
            "access_key_id": "...",
        },
        credentials={
            "secret_access_key": "...",
        },
    )
)

Updating Connections

Connection types cannot be changed once a connection is created.

CLI

To update the connection, follow the steps from Creating Connections to create a credentials file for the updated connection.

Then run the following command from

gretel connections update --id [connection-id] --from-file [updated_credentials.json]

Console

• Navigate to the Connections page using the menu item in the left sidebar.

• Go to the Connection you'd like to update and click the three vertical dots (aka overflow actions).

• Select Update Connection.

• Modify the name and/or credentials and select Save. All workflows that use this connection will automatically use the new information.

Update a connection

If you only want to edit the connection name, leave the credentials fields as they are.

SDK

Expanding on the example from Creating Connections

connection = connection_api.update_connection(
    connection_id=connection.id,
    update_connection_request=UpdateConnectionRequest(
        name="my-s3-conn",
        config={
            "access_key_id": "...",
        },
        credentials={
            "secret_access_key": "...",
        },
    )
)

Deleting Connections

CLI

gretel connections delete --id [connection-id]

Console

• On the Connections list page, select the three dots (aka overflow actions) to the right of the connection you want to delete.

• Select Delete

SDK

Expanding on the example from Creating Connections

connection_api.delete_connection(connection_id=connection.id)