Gretel Evaluate

Analyze the quality and utility of synthetic data.

Introduction

Gretel provides jobs that enable evaluation of synthetic data quality and privacy. This job is referred to as evaluate in Gretel Configurations.

You may utilize Gretel Evaluate to compare and analyze any datasets. There are no restrictions around only evaluating synthetic data that was created by Gretel.

Within the evaluate family of jobs, the following evaluation tasks are available. They can be specified within the Gretel Configuration under the task.type key.

• Validate data quality with the Synthetic Data Quality Score (SQS), task type: sqs

• Analyze performance on classification models with the classification ML Quality Score (MQS), task: classification

• Analyze performance on regression models with the regression ML Quality Score (MQS), task: regression

To see more details on each evaluate task type, please visit the Evaluate Tasks section.

Gretel Configuration

The specific evaluation task should be declared in the Gretel Configuration. By default, if a specific Evaluate task is not specified, sqs will be used.

The two configurations below are effectively identical:

schema_version: "1.0"

models:
  - evaluate:
      data_source: "__tmp__"

schema_version: "1.0"

models:
  - evaluate:
      data_source: "__tmp__"
      task:
           type: sqs

To evaluate synthetic data on classification and regression models, use:

schema_version: "1.0"

models:
  - evaluate:
      data_source: "__tmp__"
      task: classification      # or: regression
      target: "target_col"      # change this to match your data!

It is important to note that evaluatejobs are created using Gretel's Model interface. However, these models cannot be "run" so the gretel models run or SDK Record Handler creation steps will return an error if used.

Gretel evaluate jobs are single-purpose jobs, so only model creation is necessary to create an evaluate job.

CLI and SDK Usage

There are some additional considerations when running evaluate jobs through the Gretel CLI and SDK. Let's take a look at a CLI command signature below:

gretel models create --config CONFIG --in-data synthetic.csv --ref-data real-world.csv --output report-dir

Unlike other Gretel models, some of the evaluation tasks may require more than one dataset. For example, SQS requires two input datasets. The --ref-data parameter (or ref_data in the SDK) allows the use of additional datasets. The datasets can be in CSV, JSON, or JSONL format.

For evaluate, we recommend using:

• --in_data or in_data for the synthetic data under evaluation

• --ref-data or ref_data for the comparison data, such as a real-world dataset.

For SDK usage, please see the specific evaluation task that you are interested in. We have created dedicated classes in our SDK for ease of use.