Over the past few years, we've heard from many different enterprises who have tried and adopted synthetic data products from various vendors. They have found their capabilities somewhat limited in a few scenarios.

Data in the real-world enterprise environment is often extremely complex: big schemas, interconnected tables, critical database contexts and lineages that must be preserved. In some cases, lineage information must be detected automatically because it is not written down anywhere.

Building and modeling synthetic data that can act as a stand-in for real complex production data in enterprise grade settings often requires substantial capabilities, beyond generative modeling techniques alone. As this is the case, teams using immature tools run into the same eight problems.

The software does not support modeling multiple tables from disparate sources. Enterprise applications commonly use fields from 10 to 100 tables, sometimes spread across more than one database. A tool that models one table at a time can give you a convincing customer table and a convincing order table, without those two tables having a way to connect to each other.

Some tools require excessive computation and time to synthesize multiple tables. In some cases, we must provide the software with a cluster of nodes, which is too expensive and time-consuming for us. For reference, training on a single CPU machine should take minutes to an hour, and a database with 60 or more tables should fit inside that hour. When the answer to a wider schema is more hardware, the cost of test data scales with the size of your schema.

The vendor requires that we remove all NULL values before we use the product. This is labor intensive, and also limiting — what if we want to have NULL value realism in order to test for real scenarios, such as ETL pipelines? Often times, those nulls are not noise. They are a condition the application needs to handle. Stripping them removes the edge-case that breaks the ETL job downstream.

Enterprise data often contains a lot of business logic. Some vendor tools fail to capture this logic, and so their generated data is not valid. These rules are rarely written down anywhere: an amount range that depends on the currency column, or a benefits record that may only exist for premium members. Every value can be individually valid while the combination is wrong, and the application rejects a row the database considers perfect.

In one case, a vendor's tool supports multi-table generation only if the tables are joined into a single table. In another, a tool requires tables to be generated according to a specific sequence. In yet another, it is only possible to generate multiple tables for schemas with no multi-parents, which is not scalable.

The synthetic data generated by the software does not have referential integrity. This is a base requirement, not an additional feature. Without it, foreign keys point at rows that do not exist, and any test that crosses that join fails for reasons that have nothing to do with the code under test.

The software cannot create exact values in certain reference tables even when necessary. Reference tables need exact values, not plausible ones. A synthesized country code that is not in your lookup table fails validation every single time. In most cases, these tables should not be synthesized at all, and they should be carried through intact. A tool with no concept of a reference table treats them like any other and synthesizes them.

Enterprise data frequently lacks lineage, including information that connects multiple tables, such as foreign keys. In at least one case that we know of, the vendor product does not work without lineage.

If you are looking for enterprise-grade synthetic data, use these eight as your baseline when you assess a vendor. A product can look convincing in a demo and still struggle with the parts of enterprise data that matter most.