The best way to test your Gretel Hybrid deployment is to start some Gretel Jobs. You can do this quickly and easily using the Gretel CLI. If you haven't installed the Gretel CLI yet, follow our step by step instructions in our CLI and SDK environment setup guide here. Be sure to pay careful attention to any specific Gretel Hybrid instructions.
Configure Gretel CLI
The CLI and SDK environment setup guide also covers the steps necessary to configure the Gretel CLI. If you've already configured the CLI you may skip this step.
Run the below command to configure the Gretel CLI. The configure command will require step by step user inputs for each configuration element. Hit <ENTER> to accept whatever default is shown in square brackets for that configuration element.
# Let's configure the Gretel CLI
$ gretel configure
# Below is an example of the step by step input.
# Step 1. Hit enter to leave the default Gretel API endpoint.
Endpoint [https://api.gretel.cloud]:
# Step 2. Pass in the value you entered for gretel_sink_bucket_name
# in terraform.tfvars, prefixed with s3://
Artifact Endpoint [cloud]: s3://gretel-hybrid-sink
# Step 3. Enter "hybrid" for the default runner type.
Default Runner (cloud, local, hybrid) [cloud]: hybrid
# Step 4. Pass in your Gretel API key
Gretel API Key [None]: <your-api-key-here>
# Step 5. Hit enter to leave "none" for the default project, since we
# haven't created one yet!
Default Project [none]:
INFO: Configuration written to ~/.gretel/config.json. Done.
{
"endpoint": "https://api.gretel.cloud",
"artifact_endpoint": "s3://gretel-hybrid-sink",
"api_key": "grtu8d83****",
"default_project_name": null,
"default_runner": "hybrid",
"preview_features": "disabled"
}
# Let's configure the Gretel CLI
$ gretel configure
# Below is an example of the step by step input.
# Step 1. Hit enter to leave the default Gretel API endpoint.
Endpoint [https://api.gretel.cloud]:
# Step 2. Pass in the value you entered for gretel_sink_storage_container_name
# in terraform.tfvars, prefixed with azure://
Artifact Endpoint [cloud]: azure://gretel-hybrid-sink
# Step 3. Enter "hybrid" for the default runner type.
Default Runner (cloud, local, hybrid) [cloud]: hybrid
# Step 4. Pass in your Gretel API key
Gretel API Key [None]: <your-api-key-here>
# Step 5. Hit enter to leave "none" for the default project, since we
# haven't created one yet!
Default Project [none]:
INFO: Configuration written to ~/.gretel/config.json. Done.
{
"endpoint": "https://api.gretel.cloud",
"artifact_endpoint": "azure://gretel-hybrid-sink",
"api_key": "grtu5555****",
"default_project_name": null,
"default_runner": "hybrid",
"preview_features": "disabled"
}
# Let's configure the Gretel CLI
$ gretel configure
# Below is an example of the step by step input.
# Step 1. Hit enter to leave the default Gretel API endpoint.
Endpoint [https://api.gretel.cloud]:
# Step 2. Pass in the value you entered for gretel_sink_bucket_name
# in terraform.tfvars, prefixed with gs://
Artifact Endpoint [cloud]: gs://gretel-hybrid-sink
# Step 3. Enter "hybrid" for the default runner type.
Default Runner (cloud, local, hybrid) [cloud]: hybrid
# Step 4. Pass in your Gretel API key
Gretel API Key [None]: <your-api-key-here>
# Step 5. Hit enter to leave "none" for the default project, since we
# haven't created one yet!
Default Project [none]:
INFO: Configuration written to ~/.gretel/config.json. Done.
{
"endpoint": "https://api.gretel.cloud",
"artifact_endpoint": "gs://gretel-hybrid-sink",
"api_key": "grtu8d83****",
"default_project_name": null,
"default_runner": "hybrid",
"preview_features": "disabled"
}
Upload Sample Data to Source Container
You’ll need to upload the sample data files to your source data bucket or storage container. The sample CSVs are located here: sample-synthetic-healthcare.csv, patients.csv. Follow the cloud provider specific instructions below to upload the source data.
Make sure the following variables are set appropriately if they are not already exported.
export SOURCE_BUCKET=your-source-bucket-name
Upload the data to your source bucket with the following command.
Make sure the following variables are set appropriately if they are not already exported.
export SOURCE_BUCKET=your-source-bucket-name
Upload the data to your source bucket with the following command.
# Run this block of commands to upload the test CSV data to your source GCS Bucket.
wget https://raw.githubusercontent.com/gretelai/gretel-blueprints/main/sample_data/sample-synthetic-healthcare.csv
gcloud storage cp file://sample-synthetic-healthcare.csv gs://$SOURCE_BUCKET
rm -f sample-synthetic-healthcare.csv
Create a Gretel Project
One project can contain multiple models. Run the below command to create a project.