Workflows can be managed from the Console, CLI or SDK.
To manage workflows from the Console, select the Workflows tab from the left side navigation bar. This will bring you to a list of Workflows where you can view more details for each Workflow.
Using the CLI you can view commands for working with workflows by running
gretel workflows --help
Creating Workflows
Workflows can be created either from the Gretel Console or CLI.
First, create a file on your computer containing a YAML workflow config. Then run the following command
Log into the Gretel Console, and navigate to the Workflows page. Select the New Workflow button.
Next, select the project in which you'd like to create the workflow. For first time users, a Default Project will automatically be created.
Projects hold workflows, models and connections. Sharing a project will also share the entities inside it. Similarly, you cannot run a workflow if the connection is uses is in a project that is not accessible to you.
Now, select the model type. This depends on the use case. For example, if the goal is to generate synthetic data with differential privacy guarantees, choose Tabular DP.
The next step is selecting the remote data source. Since workflows are meant to be run automatically, you can't manually upload a data source. When creating and evaluating models, we recommend creating a model directly. That model can be referenced in the workflow config when it's time to operationalize your data generation.
Existing connections will show up automatically in the dropdown. If there are no connections, select New connection to define one. Add a descriptive connection name (separated by hyphens), and enter your credentials.
Provide data source and file name details. Gretel supports multiple files being processed at once. All files will create the same model type that was selected earlier in the flow.
Configure the destination. Generated data can be uploaded to the Gretel Cloud for easier access and sharing. It can also be output to a remote connection; either the same one that was configured as the data source or an entirely new one.
By default, processed files are output to the configured bucket path using the name of the data file for the model run. If you want to customize the filename or path you can modify the destination action from YAML config after completing the wizard.
With the source and destination defined, select whether the workflow should run manually or on a schedule. We provide some pre-defined schedule types, but you can also create your own schedule using a cron expression. Read more about cron expressions, or use this generator for help creating one.
Scheduled workflows don't run immediately. To test the workflow, select the Run workflow button after the workflow has been created.
The final step is reviewing the workflow configuration. For an example of a workflow config, see the section below.
The workflow configuration can be edited from this page, and the model type updated. Once the workflow has been created, it will appear on the Workflows screen. Click the workflow list item to run the workflow. Workflow run activity details will be displayed, along with detailed logs for each step.
When a workflow has successfully completed, all generated artifacts will be available in the remote destination. This includes the generated data, quality and utility reports, and log files.
import yaml
from gretel_client import create_or_get_unique_project
from gretel_client.config import get_session_config
from gretel_client.rest_v1.api.workflows_api import WorkflowsApi
from gretel_client.rest_v1.models import CreateWorkflowRequest, CreateWorkflowRunRequest
from gretel_client.workflows.logs import print_logs_for_workflow_run
session = get_session_config()
workflow_api = session.get_v1_api(WorkflowsApi)
project = create_or_get_unique_project(name="workflow-testing")
workflow_config = yaml.safe_load(
"""
name: my-helloworld-workflow
actions:
- name: run-helloworld
type: helloworld_producer
config:
names:
- ada lovelace
- alan turing
"""
)
workflow = workflow_api.create_workflow(
CreateWorkflowRequest(
name="my-helloworld-workflow",
project_id=project.project_guid,
config=workflow_config,
)
)
Workflows can be editing by navigating to that workflow, and clicking the configuration tab. Use the YAML code editor to modify workflow parameters, and select Done when completed. The new configuration will take effect for all subsequent runs of that workflow. To test the changes, select the Run workflow now button in the top right.
Building on the previous example from Creating Workflows
Workflows are organized under projects and share the same permissions as the project they are owned by.
Project Permission Mappings
Project Permission
Workflow Permissions
Read
Users can view workflows, runs, and logs.
Write
Users can create new workflows.
User can edit existing workflows.
Users can manually trigger existing workflows.
Users can delete existing workflows.
Administrator
Users can share workflows with other users.
Co-Owner
Full control.
In order to run a workflow, you must also have project writeaccess to the connections and models configured for that workflow.
Sharing Workflows
You can share a Workflow by sharing the project it is owned by. If a workflow references models or connections in a different project, be sure you have the appropriate level of access to that project.
