Benchmark Report

Check out this Benchmark report, running Gretel models on popular ML datasets, indexed by industry

You can use a Benchmark report like the one shown here to evaluate which Gretel model is best for your synthetic data goals.

For example, Gretel LSTM consistently generates synthetic data with high Synthetic Data Quality Score (SQS) on multiple types of tabular or complex data. As seen in the results below, Gretel ACTGAN is great for particularly long or wide datasets and has generally a faster runtime. If you’re looking to quickly generate lots of data, Gretel Amplify produces results in 1/10 of the time (check out the fast train and generate times!). Gretel GPT generates high-quality synthetic data for natural language datasets.

Depending on your specific goals with synthetic data or constraints, you may find particular Gretel models to be best suited for your use case. You can reference the Benchmark report below to guide how you evaluate Gretel models, or of course, try Benchmark on your datasets.

The publicly available datasets used in this results leaderboard were sourced from the following ML dataset repositories: UCI, Kaggle, and HuggingFace.

Ads, Finance, Marketing

Input Data	Size	Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
bank_marketing_large/data.csv	4.9 MB	GretelAmplify	tabular_mixed	41188	21	73	36.07	29.75	65.82
bank_marketing_large/data.csv	4.9 MB	GretelACTGAN	tabular_mixed	41188	21	85	1300.75	33.24	1333.99
bank_marketing_large/data.csv	4.9 MB	GretelLSTM	tabular_mixed	41188	21	84	317.79	401.04	718.83
bank_marketing_small/data.csv	371 KB	GretelAmplify	tabular_mixed	4521	17	80	24.41	23.81	48.22
bank_marketing_small/data.csv	371 KB	GretelACTGAN	tabular_mixed	4521	17	84	169.32	175.63	344.95
bank_marketing_small/data.csv	371 KB	GretelLSTM	tabular_mixed	4521	17	84	326.26	96.73	422.99
dow_jones_index/data.csv	89 KB	GretelAmplify	time_series	750	16	76	81.5	23.32	104.82
dow_jones_index/data.csv	89 KB	GretelACTGAN	time_series	750	16	70	221.58	129.15	350.73
dow_jones_index/data.csv	89 KB	GretelLSTM	time_series	750	16	83	424.2	64.66	488.86

E-commerce

Input data	Size	Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
bike_sales/data.csv	2.4 MB	GretelAmplify	tabular_numeric	16519	24	79	119.94	30	149.94
bike_sales/data.csv	2.4 MB	GretelLSTM	tabular_numeric	16519	24	88	911.59	249.68	1161.27
car_evaluation/data.csv	52 KB	GretelAmplify	tabular_numeric	1728	7	85	24.06	23.7	47.76
car_evaluation/data.csv	52 KB	GretelACTGAN	tabular_numeric	1728	7	77	201.19	44.5	245.69
car_evaluation/data.csv	52 KB	GretelLSTM	tabular_numeric	1728	7	87	357.66	54.02	411.68
olist_order_payments/data.csv	5.6 MB	GretelAmplify	tabular_numeric	103886	5	69	529.06	40.41	569.47
olist_order_payments/data.csv	5.6 MB	GretelLSTM	tabular_numeric	103886	5	93	4201.89	897.22	5099.11

Employment

Input data	Size	Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
data_science_job_candidates/data.csv	1.9 MB	GretelAmplify	tabular_mixed	19158	14	88	107.66	23.26	130.92
data_science_job_candidates/data.csv	1.9 MB	GretelACTGAN	tabular_mixed	19158	14	90	609.02	128.11	737.13
data_science_job_candidates/data.csv	1.9 MB	GretelLSTM	tabular_mixed	19158	14	93	358.21	276.29	634.5
ibm_employee_attrition/data.csv	274 KB	GretelAmplify	tabular_mixed	1470	37	88	24.09	20.5	44.59
ibm_employee_attrition/data.csv	274 KB	GretelACTGAN	tabular_mixed	1470	37	80	368.13	33.93	402.06
ibm_employee_attrition/data.csv	274 KB	GretelLSTM	tabular_mixed	1470	37	93	365.18	127.79	492.97

Energy, Telecom

Input data	Size	Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
energydata_complete/data.csv	11.4 MB	GretelAmplify	time_series	19735	29	74	103.68	33.09	136.77
energydata_complete/data.csv	11.4 MB	GretelLSTM	time_series	19735	29	93	1531.88	400.29	1932.17
telco_customer_churn/data.csv	1.7 MB	GretelAmplify	tabular_mixed	7043	33	82	40.84	30.17	71.01
telco_customer_churn/data.csv	1.7 MB	GretelACTGAN	tabular_mixed	7043	33	79	5911.34	55.36	5966.7
telco_customer_churn/data.csv	1.7 MB	GretelLSTM	tabular_mixed	7043	33	76	787.7	155.55	943.25

Environment, Food

Input data		Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
air_quality_uci/data.csv	822 KB	GretelAmplify	time_series	9357	15	65	96.15	52.47	148.62
air_quality_uci/data.csv	822 KB	GretelACTGAN	time_series	9357	15	62	6656.12	54.49	6710.61
air_quality_uci/data.csv	822 KB	GretelLSTM	time_series	9357	15	89	398.42	211.66	610.08
iris/data.csv	4 KB	GretelAmplify	tabular_numeric	150	5	66	87.56	23.3	110.86
iris/data.csv	4 KB	GretelACTGAN	tabular_numeric	150	5	51	140.46	170.43	310.89
iris/data.csv	4 KB	GretelLSTM	tabular_numeric	150	5	79	148.21	160.02	308.23
winequality_red/data.csv	90 KB	GretelAmplify	tabular_numeric	1599	12	82	81.94	23.68	105.62
winequality_red/data.csv	90 KB	GretelACTGAN	tabular_numeric	1599	12	61	76.14	43.87	120.01
winequality_red/data.csv	90 KB	GretelLSTM	tabular_numeric	1599	12	89	221.92	54.69	276.61
winequality_white/data.csv	281 KB	GretelAmplify	tabular_numeric	4898	12	88	24.65	23.27	47.92
winequality_white/data.csv	281 KB	GretelACTGAN	tabular_numeric	4898	12	81	139.03	33.15	172.18
winequality_white/data.csv	281 KB	GretelLSTM	tabular_numeric	4898	12	91	287.84	76.4	364.24

Government

Input data	Size	Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
portuguese_election_data/data.csv	3 MB	GretelAmplify	tabular_numeric	21643	28	52	31.33	107.04	138.37
portuguese_election_data/data.csv	3 MB	GretelACTGAN	tabular_numeric	21643	28	72	928.15	128.73	1056.88
portuguese_election_data/data.csv	3 MB	GretelLSTM	tabular_numeric	21643	28	81	455.56	327.19	782.75
adult/data.csv	3.6 MB	GretelAmplify	tabular_mixed	32561	15	85	213.54	58.46	272
adult/data.csv	3.6 MB	GretelACTGAN	tabular_mixed	32561	15	87	965.31	128.03	1093.34
adult/data.csv	3.6 MB	GretelLSTM	tabular_mixed	32561	15	94	667.21	615.08	1282.29

Healthcare

Input data	Size	Model	Datatype	Rows	Cols	SQS	Train time (s)	Generate time (s)	Total time (s)
processed_cleveland_heart_disease_uci/data.csv	18 KB	GretelAmplify	tabular_numeric	303	14	83	35.87	23.01	58.88
processed_cleveland_heart_disease_uci/data.csv	18 KB	GretelACTGAN	tabular_numeric	303	14	70	66.97	33.48	100.45
processed_cleveland_heart_disease_uci/data.csv	18 KB	GretelLSTM	tabular_numeric	303	14	91	221.56	54.26	275.82
breast_cancer_wisconsin/data.csv	19 KB	GretelAmplify	tabular_numeric	699	11	55	23.89	23.55	47.44
breast_cancer_wisconsin/data.csv	19 KB	GretelACTGAN	tabular_numeric	699	11	56	67.4	203.11	270.51
breast_cancer_wisconsin/data.csv	19 KB	GretelLSTM	tabular_numeric	699	11	83	206.88	64.73	271.61

Large Datasets

Input data	Size	Model	DataType	Rows	Columns	SQS	Train time (sec)	Generate time (sec)	Total time (sec)
covertype	75 MB	GretelAmplify	tabular_numeric	581012	55	91	238.24	60.64	298.88
covertype	75 MB	GretelACTGAN	tabular_numeric	581012	55	92	5870.86	1004.67	6875.53
covertype	75 MB	GretelLSTM	tabular_numeric	581012	55	83	7547.14	2317.74	9864.88
duck_duck_geese	311 MB	GretelAmplify	tabular_numeric	27000	1349	53	1962.59	593.78	2556.37
duck_duck_geese	311 MB	GretelACTGAN	tabular_numeric	27000	1349	75	55879.28	425.86	56305.14
kdd_cup_1999	743 MB	GretelACTGAN	tabular_mixed	4898430	42	87	17408.25	3986.04	21394.29
pems_sf	421 MB	GretelAmplify	tabular_numeric	63360	967	68	3103.19	1247.72	4350.91
phoneme_spectra	154 MB	GretelAmplify	tabular_numeric	1446956	15	78	402.33	256.85	659.18
phoneme_spectra	154 MB	GretelACTGAN	tabular_numeric	1446956	15	74	5263.59	1741.28	7004.87
phoneme_spectra	154 MB	GretelLSTM	tabular_numeric	1446956	15	86	5099.49	1777.7	6877.19
poker_hand_test	24 MB	GretelAmplify	tabular_numeric	1000000	11	92	138.71	76.8	215.51
poker_hand_test	24 MB	GretelACTGAN	tabular_numeric	1000000	11	93	2207.99	1047.02	3255.01
poker_hand_test	24 MB	GretelLSTM	tabular_numeric	1000000	11	98	1646.91	1018.37	2665.28
poker_hand_train	614 KB	GretelAmplify	tabular_numeric	25010	11	94	28.55	23.7	52.25
poker_hand_train	614 KB	GretelACTGAN	tabular_numeric	25010	11	91	852.35	33.23	885.58
poker_hand_train	614 KB	GretelLSTM	tabular_numeric	25010	11	99	459.17	276.44	735.61
record_linkage	262 MB	GretelAmplify	tabular_numeric	5749132	12	75	420.3	179.93	600.23
record_linkage	262 MB	GretelACTGAN	tabular_numeric	5749132	12	75	12476.42	4935.9	17412.32
rossman_store_sale	38 MB	GretelAmplify	tabular_mixed	1017209	9	71	242.79	68.59	311.38
rossman_store_sale	38 MB	GretelLSTM	tabular_mixed	1017209	9	86	2413.37	1120.68	3534.05
rossman_store_sale	38 MB	GretelACTGAN	tabular_mixed	1017209	9	74	3107.57	1540.63	4648.2