Test Your Deployment

The best way to test your Gretel Hybrid deployment is to start some Gretel Jobs. You can do this quickly and easily using the Gretel CLI. If you haven't installed the Gretel CLI yet, follow our step by step instructions in our CLI and SDK environment setup guide here. Be sure to pay careful attention to any specific Gretel Hybrid instructions.

Configure Gretel CLI

The CLI and SDK environment setup guide also covers the steps necessary to configure the Gretel CLI. If you've already configured the CLI you may skip this step.

Run the below command to configure the Gretel CLI. The configure command will require step by step user inputs for each configuration element. Hit <ENTER> to accept whatever default is shown in square brackets for that configuration element.

# Let's configure the Gretel CLI
$ gretel configure

# Below is an example of the step by step input.

# Step 1. Hit enter to leave the default Gretel API endpoint.
Endpoint [https://api.gretel.cloud]:

# Step 2. Pass in the value you entered for gretel_sink_bucket_name 
# in terraform.tfvars, prefixed with s3://
Artifact Endpoint [cloud]: s3://gretel-hybrid-sink

# Step 3. Enter "hybrid" for the default runner type.
Default Runner (cloud, local, hybrid) [cloud]: hybrid

# Step 4. Pass in your Gretel API key
Gretel API Key [None]: <your-api-key-here>

# Step 5. Hit enter to leave "none" for the default project, since we
# haven't created one yet!
Default Project [none]:

INFO: Configuration written to ~/.gretel/config.json. Done.
{
    "endpoint": "https://api.gretel.cloud",
    "artifact_endpoint": "s3://gretel-hybrid-sink",
    "api_key": "grtu8d83****",
    "default_project_name": null,
    "default_runner": "hybrid",
    "preview_features": "disabled"
}

# Let's configure the Gretel CLI
$ gretel configure

# Below is an example of the step by step input.

# Step 1. Hit enter to leave the default Gretel API endpoint.
Endpoint [https://api.gretel.cloud]:

# Step 2. Pass in the value you entered for gretel_sink_storage_container_name
# in terraform.tfvars, prefixed with azure://
Artifact Endpoint [cloud]: azure://gretel-hybrid-sink

# Step 3. Enter "hybrid" for the default runner type.
Default Runner (cloud, local, hybrid) [cloud]: hybrid

# Step 4. Pass in your Gretel API key
Gretel API Key [None]: <your-api-key-here>

# Step 5. Hit enter to leave "none" for the default project, since we
# haven't created one yet!
Default Project [none]:

INFO: Configuration written to ~/.gretel/config.json. Done.
{
    "endpoint": "https://api.gretel.cloud",
    "artifact_endpoint": "azure://gretel-hybrid-sink",
    "api_key": "grtu5555****",
    "default_project_name": null,
    "default_runner": "hybrid",
    "preview_features": "disabled"
}

# Let's configure the Gretel CLI
$ gretel configure

# Below is an example of the step by step input.

# Step 1. Hit enter to leave the default Gretel API endpoint.
Endpoint [https://api.gretel.cloud]:

# Step 2. Pass in the value you entered for gretel_sink_bucket_name 
# in terraform.tfvars, prefixed with gs://
Artifact Endpoint [cloud]: gs://gretel-hybrid-sink

# Step 3. Enter "hybrid" for the default runner type.
Default Runner (cloud, local, hybrid) [cloud]: hybrid

# Step 4. Pass in your Gretel API key
Gretel API Key [None]: <your-api-key-here>

# Step 5. Hit enter to leave "none" for the default project, since we
# haven't created one yet!
Default Project [none]:

INFO: Configuration written to ~/.gretel/config.json. Done.
{
    "endpoint": "https://api.gretel.cloud",
    "artifact_endpoint": "gs://gretel-hybrid-sink",
    "api_key": "grtu8d83****",
    "default_project_name": null,
    "default_runner": "hybrid",
    "preview_features": "disabled"
}

Upload Sample Data to Source Container

You’ll need to upload the sample data files to your source data bucket or storage container. The sample CSVs are located here: sample-synthetic-healthcare.csv, patients.csv. Follow the cloud provider specific instructions below to upload the source data.

Make sure the following variables are set appropriately if they are not already exported.

export SOURCE_BUCKET=your-source-bucket-name

Upload the data to your source bucket with the following command.

wget https://raw.githubusercontent.com/gretelai/gretel-blueprints/main/sample_data/sample-synthetic-healthcare.csv
aws s3 cp sample-synthetic-healthcare.csv s3://$SOURCE_BUCKET
rm -f sample-synthetic-healthcare.csv

Make sure the following variables are set appropriately if they are not already exported.

export SOURCE_CONTAINER_NAME=gretel-hybrid-source
export STORAGE_ACCOUNT_NAME=gretelhybrid

Upload the data to your source storage container with the following command.

wget https://raw.githubusercontent.com/gretelai/gretel-blueprints/main/sample_data/sample-synthetic-healthcare.csv
az storage blob upload --account-name "$STORAGE_ACCOUNT_NAME" \
    -c "$SOURCE_CONTAINER_NAME" \
    -f "sample-synthetic-healthcare.csv"
rm -f sample-synthetic-healthcare.csv

Make sure the following variables are set appropriately if they are not already exported.

export SOURCE_BUCKET=your-source-bucket-name

Upload the data to your source bucket with the following command.

# Run this block of commands to upload the test CSV data to your source GCS Bucket.
wget https://raw.githubusercontent.com/gretelai/gretel-blueprints/main/sample_data/sample-synthetic-healthcare.csv
gcloud storage cp file://sample-synthetic-healthcare.csv gs://$SOURCE_BUCKET
rm -f sample-synthetic-healthcare.csv

Create a Gretel Project

One project can contain multiple models. Run the below command to create a project.

GRETEL_PROJECT="Gretel-Hybrid-Test-$RANDOM"
gretel projects create --name "$GRETEL_PROJECT" --project-type hybrid --display-name "Gretel Hybrid Test"

Run a Test Job

Follow the below examples to run a test model training job.

CPU based example

gretel models create --config transform/transform_v2_hybrid_example \
  --in-data s3://$SOURCE_BUCKET/patients.csv \
  --runner hybrid \
  --project $GRETEL_PROJECT

GPU based example

gretel models create --config synthetics/tabular-actgan \
    --in-data s3://$SOURCE_BUCKET/sample-synthetic-healthcare.csv \
    --runner hybrid \
    --project $GRETEL_PROJECT

CPU based example

gretel models create --config transform/transform_v2_hybrid_example \
  --in-data azure://$SOURCE_CONTAINER_NAME/patients.csv \
  --runner hybrid \
  --project $GRETEL_PROJECT

GPU based example

gretel models create --config synthetics/tabular-actgan \
    --in-data azure://$SOURCE_CONTAINER_NAME/sample-synthetic-healthcare.csv \
    --runner hybrid \
    --project $GRETEL_PROJECT

CPU based example

gretel models create --config transform/transform_v2_hybrid_example \
  --in-data gs://$SOURCE_BUCKET/patients.csv \
  --runner hybrid \
  --project $GRETEL_PROJECT

GPU based example

gretel models create --config synthetics/tabular-actgan \
    --in-data gs://$SOURCE_BUCKET/sample-synthetic-healthcare.csv \
    --runner hybrid \
    --project $GRETEL_PROJECT

PreviousGCP Setup NextContainer Image Cache Support

Last updated 1 month ago

Was this helpful?