Evaluating the efficacy and quality of synthetic data.


Gretel provides jobs that enable evaluation of synthetic data quality and privacy. This job (or model type) is referred to as evaluate in Gretel Configurations.
You may utilize Gretel Evaluate to compare and analyze any datasets. There are no restrictions around only evaluating synthetic data that was created by Gretel.
Within the evaluate family of jobs, the following evaluation tasks are available. They can be specified within the Gretel Configuration under the task.type key.
  • Synthetic Quality Score (SQS), task type: sqs
To see more details on each evaluation task type, please visit the Evaluation Tasks section.

Gretel Configuration

The specific evaluation task should be declared in the Gretel Configuration. By default, if a specific evaluation task is not specified, SQS will be used by default.
The two configurations below are effectively identical:
schema_version: 1.0
- evaluate:
data_source: "__tmp__"
schema_version: 1.0
- evaluate:
data_source: "__tmp__"
type: sqs
It is important to note that evaluate jobs are created using Gretel's Model interface. However, these models cannot be "run" so the gretel models run or SDK Record Handler creation steps will return an error if used.
Gretel evaluate jobs are single-purpose jobs, so only model creation workflows should be used.

CLI and SDK Usage

There are some additional considerations when running evaluate jobs through the Gretel CLI and SDK. Let's take a look at a CLI command signature below:
$ gretel models create --config CONFIG --in-data synthetic.csv --ref-data real-world.csv --output report-dir
Unlike other Gretel models, some of the evaluation tasks may require more than one dataset. For example, SQS requires two input datasets. The --ref-data parameter (or ref_data in the SDK) allows the use of additional datasets.
For evaluate, we recommend using:
  • --in_data or in_data for the synthetic data under evaluation
  • --ref-data or ref_data for the comparison data, such as a real-world dataset.
For SDK usage, please see the specific evaluation task that you are interested in. We have created dedicated classes in our SDK for ease of use.