Classification
Evaluate synthetic data vs. real-world data on classification models
The
classification Evaluate task will generate a Gretel Synthetic Data Utility Report. Customers frequently ask whether synthetic data is of high enough quality to train downstream machine learning tasks. Classifiers, for example, require highly accurate data before they can be usefully deployed.
The Gretel Evaluate Classification task uses the open source AutoML PyCaret library under the hood to evaluate the quality of your generated synthetic data on commonly used downstream machine learning classifiers, and gives you the results in an easy-to-understand HTML report.
You can kick off this evaluation directly in the Gretel Console. Start with this workflow:
Generate synthetic data + evaluate ML performanceThis workflow includes a sample dataset (the publicly available bank marketing dataset) and the default blueprint:
- Gretel LSTM model to generate synthetic data
classificationEvaluate task with default parameters
evaluate:
task: classification
target: "y" ##### IMPORTANT! RENAME TO THE LABEL/PREDICTION COLUMN NAME OF YOUR DATA ######
# holdout: null # Default train-test split for model training = 0.2
# metric: null # Default metric = "acc"
# models: null # By default all available models are trained
You can leave the config as is and simply click "Begin training" or edit the configuration with the synthetic model and optional
classification parameters best suited for your use case.
By default, all models will be used in the classifier model training. You can select specific models to use by passing in a list of strings from the following set:
# All classification models
classification_models = [
"lr",
"knn",
"nb",
"dt",
"svm",
"rbfsvm",
"gpc",
"mlp",
"ridge",
"rf",
"qda",
"ada",
"gbc",
"lda",
"et",
"xgboost",
"lightgbm",
"dummy"
]
If you want to change the metric that the classifiers will use to optimize for, you can select one metric from
classification_metrics below. The default metric is "acc" (accuracy).# Select a metric
classification_metrics = [
"acc",
"auc",
"recall",
"precision",
"f1",
"kappa",
"mcc"
]
You can use the
classification Evaluate task in two ways: 1. As a parameter of a Gretel synthetics model, or
2. Compare two datasets directly: a synthetic dataset and a real-world dataset
Here's a basic example generating synthetic data using Gretel ACTGAN and the real-world bank marketing dataset, then adding
classification evaluation to create the Data Utility Report:from gretel_client.helpers import poll
from gretel_client.projects.models import read_model_config
from gretel_client.projects import create_or_get_unique_project
# Create a project with a name that describes this use case
project = create_or_get_unique_project(name="bank-classification-docs-example")
# Import the bank_marketing_small dataset from Gretel's public S3 bucket
dataset_path = "https://gretel-datasets.s3.amazonaws.com/bank_marketing_small/data.csv"
# Add Evaluate task to Gretel ACTGAN config
config = read_model_config("synthetics/tabular-actgan")
config["models"][0]["actgan"]["evaluate"] = {
"task": "classification",
"target": "y"
}
You can then run the model and save the report using:
## Train and run the model
model = project.create_model_obj(
model_config=config,
data_source=dataset_path
)
model.submit_cloud()
poll(model)
# Save all artifacts
model.download_artifacts("/tmp")
Even when using the Evaluate SDK, you can find model details and report download options in the Gretel Console -- simply navigate to the
bank-marketing-classification-example project.
If you already have generated synthetic data in the form of a CSV, JSON(L) or Pandas Dataframe, you can also use this Evaluate task to analyze the two datasets.
The Gretel SDK provides Python classes specifically to run reports. The
DownstreamClassificationReport() class uses evaluate with classification task to generate a Data Utility Report. A basic usage is below:# Use Evaluate SDK
from gretel_client.evaluation.downstream_classification_report import DownstreamClassificationReport
# Create a project
from gretel_client.projects import create_or_get_unique_project
project = create_or_get_unique_project(name="evaluate-bank-classification-example-2")
# Params
# NOTE: These data sources may also be Pandas DataFrames!
data_source = "synthetic.csv"
ref_data = "real.csv"
# Target to predict, REQUIRED for evaluate model
target = 'y'
# Default holdout value
# test_holdout = 0.2
# Supply a subset if you do not want all of these, default is to use all of them
# models = classification_models
# Metric to use for ordering results, default is "acc" (Accuracy) for classification
# metric = "acc"
# Create a downstream classification report
evaluate = DownstreamClassificationReport(
project=project,
target=target,
data_source=data_source,
ref_data=ref_data,
# holdout=test_holdout,
# models=models,
# metric=metric,
# runner_mode="cloud",
)
The Evaluate task creates a Data Utility Report with the results of the analysis. You'll see a high-level ML Quality Score (MQS) which gives you an at-a-glance understanding of how your synthetic dataset performed. For more info about the report, checkout this page about each section.
You can view logs both in the SDK environment or go to the project in the Gretel Console to follow along with the model training progress and download the results of the evaluation.
Last modified 5mo ago