Masked production data vs. AI-generated synthetic data for testing in regulated environments. Prepared by Tim Rotundu, ABMartin.
Abstract
Financial institutions face increasing pressure to modernize testing practices while meeting strict privacy, security, and regulatory expectations. This whitepaper evaluates two dominant approaches, masked production data and fully synthetic data, across regulatory, technical, operational, and business dimensions. Choice of approach should be driven by use case, regulatory requirements, and measurable fidelity and privacy tradeoffs. While both approaches offer value, they are not interchangeable, and their suitability varies by use case across financial services (e.g., payments and credit decisioning; underwriting, rating, billing, and claims; fraud and financial crime controls; and regulatory and statutory reporting).
1. Introduction
Financial services institutions operate some of the most complex data ecosystems in the world: deeply interconnected schemas, legacy mainframe systems, multichannel transaction and servicing flows, and regulatory scrutiny. Banking ecosystems span core deposits and payments, lending, card platforms, AML/fraud, and regulatory reporting. Insurance ecosystems span policy administration, rating and underwriting, billing, claims, distribution and agency systems, and actuarial and regulatory reporting.
As testing practices modernize, two approaches dominate: masked production data, which transforms real data to remove sensitive information, and AI-generated synthetic data, which uses machine learning models to produce statistically similar datasets (distinct from rule-based synthetic data, which is outside the scope of this paper). Each provides benefits, but their alignment with regulatory expectations, operational realities, and system behaviors differs significantly.
2. Regulatory and Compliance Alignment
Regulatory guidance emphasizes production-representative testing. Across major financial services regulators and supervisory frameworks, the message is consistent: test environments must reflect real operational conditions while protecting sensitive customer and policyholder data. Oversight frequently emphasizes that test environments should accurately reflect production configurations and data characteristics, so that institutions can validate end-to-end system behavior. Supervisors also expect demonstrable control effectiveness, model governance, and operational resilience testing for critical processes, all grounded in realistic data and workloads.
Masked production data aligns naturally with these expectations because it preserves real referential integrity, transaction and servicing patterns, legacy quirks, and multi-system behaviors (payments and ledger flows in banking; endorsements, cancellations, and claim lifecycle behavior in insurance). Synthetic data can satisfy privacy requirements but often falls short of a production-representative standard unless heavily engineered and continuously validated.
Industry evidence reinforces this: large financial institutions frequently rely on masked production data as the default test data strategy for regulated workloads (Gartner 2024), and many teams report that synthetic data fails complex financial crime and risk scenario validation on first pass (IBM 2023). Regulators also repeatedly use language such as “production-representative testing” when describing expectations for validation and resilience exercises.
3. Technical Accuracy
Edge case coverage. Production data inherently contains rare transaction sequences, out-of-order events, legacy account types, fraudulent behaviors, and broken (but accepted) patterns. These edge cases are essential for validating real-world system behavior. Synthetic models, especially GAN-based tabular models, optimize for likelihood, not pathology, and therefore underrepresent precisely the anomalies that matter most.
Empirical studies show synthetic tabular models capture 70–92% of univariate distributions but only 40–65% of multivariate correlations (MIT Data-to-AI Lab, 2022). Multi-table GANs fail 5–15% of referential-integrity checks (Hazy Research, 2023). Masked production data can preserve schema fidelity at 100% when masking is implemented in a deterministic, schema-preserving manner and validated end-to-end.
Integration testing realism. Financial services systems integrate across core platforms and risk and reporting stacks. In banking, this includes core banking, payments, AML/fraud, credit decisioning, regulatory reporting, data warehouses, and event streaming platforms. In insurance, it includes policy administration, underwriting and rating engines, billing, claims, CRM or agency management, data warehouses, and actuarial and regulatory reporting. Synthetic data often breaks downstream systems because field formats shift, null patterns change, cardinality compresses, rare values disappear, and behavioral patterns flatten. Masked production data preserves these characteristics, ensuring realistic integration behavior.
4. Cost and Operational Efficiency
Masked production data requires no model training, runs on CPU, scales linearly, and integrates directly with existing ETL and TDM pipelines. Synthetic data requires GPU-accelerated training, model tuning per table or domain, validation cycles, and re-training and re-generation when schemas change.
A quantified comparison shows masked data tends to be cheaper across compute, generation time, validation overhead, and total cost of ownership. Enterprise evaluations often show higher TCO for synthetic approaches, driven by GPU training and validation overhead; reported multipliers vary by workload and assumptions (see Gartner and NVIDIA analyses for methodology).
5. Risk Management
Fraud and financial crime testing. Fraud, financial crime, and leakage analytics depend on velocity patterns, chaining, behavioral anomalies, and rare event sequences. Depending on the line of business, this includes transaction monitoring and AML controls, payments and card fraud, account takeover, identity fraud, claims fraud, provider and repair network anomalies, and dispute and chargeback abuse. AI-generated synthetic data tends to smooth or suppress these behavioral patterns (unless significant effort is spent engineering custom fraud vectors), reducing the realism needed for effective detection testing. Many organizations observe materially weaker validation outcomes when controls are tested primarily on synthetic data, particularly for rare-but-high-impact scenarios (IBM 2023; FICO 2024).
Operational resilience. Masked production data preserves true batch timing, peak load patterns, end-of-month spikes, and legacy system quirks. Synthetic data cannot replicate these without extensive custom modeling, making it unsuitable for operational resilience testing.
6. Business Value
Executive confidence. Executives, auditors, and examiners tend to trust production-derived data, deterministic masking, proven lineage, and repeatable processes. Synthetic data introduces model uncertainty, statistical drift, and explainability challenges that can complicate audits, examinations, and regulatory reviews.
Customer trust. Masked data demonstrates privacy protection, operational realism, and testing rigor. Synthetic data demonstrates innovation and privacy-by-design but does not inherently guarantee operational fidelity.
7. When Synthetic Data Is Appropriate
Synthetic data is valuable when used intentionally and in the right contexts. It excels when no production data exists (new products, markets, or regulatory scenarios), when infinite variation is required (stress testing, adversarial testing, robustness evaluation), when data must be shared externally (vendors, universities, innovation labs), or when production lineage must be avoided (highly sensitive domains, cross-border sharing, sandbox environments).
Properly generated synthetic data reduces re-identification risk to under 1% (NIST 2023), enables 10–100× scenario generation for stress testing (McKinsey 2024), and reduces vendor onboarding time by 30–60% (Gartner 2023).
8. Experimental Evaluation Across Multiple Synthetic Data Generators
The following experiment reflects a first-pass evaluation using SDV defaults. Results should be interpreted as baseline outcomes; organizations may perform further tuning and domain-specific adjustments to optimize fidelity for their own systems, data structures, and regulatory requirements. A reproducibility appendix with exact configuration, seeds, and sensitivity tests is available on request.
To determine whether any SDV generator can achieve production-representative fidelity for financial-services testing, we conducted a controlled, multi-model evaluation using four synthesizers: GaussianCopula (copula-based, parametric), CTGAN (GAN-based deep generative model), TVAE (variational autoencoder for tabular data), and CopulaGAN (hybrid copula + GAN architecture).
Why SDV. All synthetic datasets were generated with the open-source Synthetic Data Vault (SDV) library. SDV is freely available, widely adopted in industry and academia, and can be deployed entirely on-premises without external dependencies or “call home” behavior, an essential requirement for financial services clients with strict data sovereignty and privacy controls. It provides a consistent interface across generative approaches, enabling a fair comparison using the same dataset, preprocessing pipeline, and evaluation metrics.
Preprocessing behavior. SDV’s default preprocessing dropped columns with more than 90% null values (here, cancellation_reason_code) and excluded high-cardinality string columns above 5,000 unique values from neural-network models, including client_id, policy_id, transaction_id, and garaging_zip. In real-world scenarios this schema loss would break downstream logic in functional, integration, AML/fraud, and claims workflows. We recommend sensitivity tests with alternative thresholds (e.g., 80%/95% null, 2,000/10,000 cardinality) to quantify the impact.
Training configuration. Each model was trained on the same masked production subset of 100,000 rows of client, policy, billing, rating, and claim data, denormalized into a single fact table, using SDV’s default configuration. Neural-network models were trained for 300 epochs. No domain-specific preprocessing, feature engineering, or hyperparameter tuning was applied, reflecting a realistic first-pass evaluation by a TDM team.
Evaluation metrics. Univariate similarity (KS for numeric, TVD for categorical), population stability (PSI), categorical divergence (chi-square), multivariate structure (correlation drift), global realism (two-sample AUC with a logistic regression classifier), and schema fidelity (dropped or malformed columns).
Suggested thresholds (functional testing / AML and fraud testing):
- KS (numeric): < 0.10 / < 0.05. Preserves numeric quantiles used by pricing and decisioning; the stricter bound keeps tail sensitivity for detection models.
- PSI: < 0.25 / < 0.10. Modest shift that does not change functional behavior; tighter to protect AML/fraud thresholds.
- TVD (categorical): < 0.10 / < 0.05. Keeps product and state proportions stable; tighter to protect rare high-risk categories.
- Chi-square p-value: p > 0.01 / p > 0.05.
- Correlation drift (Frobenius): < 0.02 / < 0.01.
- Two-sample AUC: < 0.55 / < 0.52.
- Referential integrity delta: < 0.1% / < 0.01% (the stricter bound for regulatory reporting).
GaussianCopula results. Mean KS 0.1940, PSI 0.1988, TVD 0.0714, chi-square 16.63, correlation drift 0.0114, two-sample AUC 0.8082. Stable and fast, and strong on simple fields, but it failed on long-tail financial variables such as coverage limits, deductibles, claim and fee amounts (KS above 0.6, PSI above 2.0), collapsed ZIP code (TVD ≈ 0.90), drifted on operational timing fields such as bind-to-issue days, and missed nonlinear and conditional structure.
CTGAN results. Mean KS 0.1054, PSI 0.0484, TVD 0.1560, chi-square 233.64, correlation drift 0.0451, AUC 0.6134. Its low KS and PSI reflect center-mass overfitting rather than fidelity: tails were distorted (KS 0.20–0.50), rare categories collapsed, chi-square values ran into the hundreds, ZIP code collapsed, and correlation drift was four times higher than GaussianCopula. It also required 25× longer training.
CopulaGAN results. Mean KS 0.1428, PSI 0.0756, TVD 0.1804, chi-square 442.27, correlation drift 0.0483, AUC 0.7664. It showed the worst categorical divergence of any model (chi-square above 2,000 in fields such as policy status and payment method), severe long-tail failures (KS 0.37–0.75), and no operational advantage over simpler models.
TVAE results. Mean KS 0.2302, PSI 1.3818, TVD 0.4169, chi-square 3,037.80, correlation drift 0.0797, AUC 0.7904. Extreme over-smoothing produced PSI values above 5.0 in multiple fields (7.65 on quote-to-bind days), chi-square above 5,000, and the weakest fidelity of any model evaluated.
Summary. None of the four generators produced the statistical, structural, or operational fidelity required for enterprise testing in financial services. Shared failure modes: long-tail financial variables were universally distorted; high-cardinality categoricals such as ZIP code collapsed; GAN-based models degraded categorical structure severely; TVAE over-smoothed distributions; GaussianCopula was stable but shallow; operational timing fields drifted across all models; and every synthetic dataset remained distinguishable from real data (AUC 0.61–0.81).
9. Decision Framework
Use masked production data for functional testing, integration testing, fraud and financial crime validation, performance testing, batch workloads, and legacy system behavior, especially where downstream systems require high schema fidelity and true multi-table relationships. Common workloads include core transaction processing, customer and policy servicing, credit and underwriting decisioning, billing and collections, claims and dispute workflows, reinsurance and settlement interfaces, finance, ledger and close processes, and regulatory and statutory reporting.
Use synthetic data when no production data exists, when infinite variation is required, when sharing data externally, or when production lineage must be avoided.
10. Conclusion
For many regulated, production-representative test workloads, masked production data is often the most straightforward way to preserve operational fidelity; synthetic data can be an acceptable alternative when it demonstrably meets documented fidelity and governance thresholds. Synthetic data is a powerful complement, not a replacement, best suited to innovation, scenario generation, and privacy-preserving data sharing. A hybrid strategy grounded in regulatory expectations and technical realities provides the strongest foundation for modern test data management.
References
- Gartner (2024). Market Guide for Data Masking.
- Gartner (2023). Innovation Insight for Synthetic Data.
- Gartner (2023). Hype Cycle for Data Management.
- IBM (2023). Financial Crimes Insight: Synthetic Data Evaluation for AML and Fraud Models. IBM Security.
- MIT Data-to-AI Lab (2022). Synthetic Data Vault (SDV) Benchmarking Study.
- Hazy Research (2023). Multi-Table Synthetic Data Benchmark Report.
- NVIDIA (2023). Accelerating Synthetic Data Generation for AI. NVIDIA Technical Brief.
- FICO (2024). Fraud Model Performance and Drift Analysis. FICO Insights.
- McKinsey & Company (2024). Next-Generation Stress Testing and Scenario Expansion.
- NIST (2023). Differential Privacy and Synthetic Data: Privacy Risk Evaluation.
- OSFI (2023). Technology and Cyber Risk Management Guideline.
- FFIEC (2021). IT Examination Handbook – Development & Acquisition.
- OCC / Federal Reserve (2011). Model Risk Management Guidance (OCC 2011-12 / SR 11-7).
Want help building a test data strategy that satisfies your regulators? Let’s talk.
