Datagaps Test Data Manager (TDM) is an AI-powered synthetic data tool that learns patterns from sample data to generate privacy-safe test data for Microsoft Fabric lakehouses and warehouses, masking PII automatically and reducing work that could take weeks to under an hour.
Key Takeaways
- Manually copying or creating test data breaks down at 30 to 40 or more tables per testing cycle.
- TDM has two components: Data Models configure generation, and Datasets generate and load the data.
- AI auto-detects entities, treats classified columns as PII or confidential, and masks them during generation.
- Sample data is a prerequisite; by default TDM takes 5,000 rows from the connected source.
- Generating 100 or 1,000 rows completes in about five seconds, with multiple tables processed in parallel.
To generate synthetic test data for Microsoft Fabric, configure a Data Model in Datagaps Test Data Manager (TDM) — connect your lakehouse or warehouse, let AI detect and mask entities, load sample data and confirm target data types — then use a Dataset to generate the volume you need and load it into tables or files.
Why is test data for Microsoft Fabric pipelines hard to manage?
In Microsoft Fabric, pipelines and notebooks are only as good as the data used to test them. In a typical enterprise setting, developers work with minimal or dummy data, and QA teams manually copy production data or create new entries for staging. In large setups with 30 to 40 or more tables per testing cycle, this becomes unsustainable fast, especially when PII masking and relational integrity are also in scope.
Datagaps Test Data Manager (TDM) addresses this directly. Its AI capabilities generate synthetic data that covers all production-level nuances without manual effort, reducing what could take weeks to under an hour.
How does Datagaps TDM generate synthetic data for Microsoft Fabric?
Datagaps TDM has two core components: Data Models and Datasets.
Data Models are where you configure the data you want to generate for each table. This includes masking data, fetching sample data for generation, defining the target data type, and all other configuration requirements.
Datasets use the configured data model and generate data using AI/ML capabilities. You define the volume of data to be generated and then choose to either save it to an external location or load it directly into tables for testing.
How do you configure a TDM data model for Microsoft Fabric tables?
Connect the Fabric lakehouse or warehouse
For demonstration purposes, two schemas were connected into the data model, containing 5 to 10 dimension and fact tables, each with approximately 20 to 30 columns. When you connect your Fabric data source, you connect your specific lakehouse or warehouse. The moment a data source is connected, Datagaps fetches relevant metadata and automatically reflects it in the interface.
How are entities classified and PII masked?
There is an additional column called Entity, which is used to define the type of a column and whether it needs to be masked. When any column is classified as one of the defined entities, Datagaps treats that data as PII or confidential and masks it during generation. Datagaps already supports GDPR, CCPA, and HIPAA-aligned workflows, which means data security concerns are addressed at the platform level.
Entity detection can be done manually, or Datagaps can handle it automatically using AI. There is a built-in option to auto-detect entities for your tables.
When auto-detect entities is used, Datagaps applies its internal AI/ML models to classify each column into one of its predefined entity categories. Columns containing names, payment details, addresses, date-time values, and similar data are auto-classified without requiring manual intervention. This is a particularly powerful feature when connecting tables for the first time and configuring them at scale. The platform can generate millions of records across formats with built-in masking.
Why is sample data required?
The only prerequisite is that some sample data must be loaded before using this feature. To auto-detect entities and generate synthetic data, existing data is required so the system can identify patterns in those values. You can filter that sample however you choose, whether using a specific set of values or a random selection. This sample enables the AI to understand the patterns needed for accurate synthetic data generation.
How do you preserve referential integrity between tables?
For projects with relationships between tables, these need to be accounted for carefully. A foreign key, for example, must be mapped correctly and must be available in its source table. Otherwise, the generated data will lack referential integrity. Datagaps allows you to define these relationships and map columns to their parent column in another table.
How much sample data does TDM use?
By default, Datagaps takes 5,000 rows of sample data from the connected source. This can be increased or decreased after initial testing. It also allows you to add a WHERE clause to filter specific data for sample generation, which is useful when testing for a particular region or segment.
How are target data types defined?
Once that configuration is complete, the next step is defining the quality of your synthetic data. This is where you define the target data types for generation. For entity columns, there are predefined data types that Datagaps refers to during generation. This too does not need to be done manually. The auto-recommend feature assigns entities and correct target data types for each table. All that is required is a review to confirm accuracy.
When you generate your sample data, you will notice that certain columns are masked. These are the same columns for which an entity was defined in the configuration step.
What is Fake Data in Datagaps TDM?
The real capability of synthetic data generation becomes apparent at this step. For columns containing names, payment details, or other entity types, the system automatically classifies the target type and sets the data up as realistic fake data. This is what Datagaps refers to as Fake Data.
Fake data consists of AI or machine-generated values defined across multiple categories. Datagaps maintains a predefined list of categories aligned with real-world data, and it classifies columns into one of those categories before generation. While the AI handles classification accurately in most cases, it is strongly recommended to review the target type of each column at this step. Verify that a column is not misclassified, for example, as an address when it actually contains a name.
Almost any category of data you can think of is predefined within TDM, including blood group, card details, age, email, contact information, and more. All of them can be mapped correctly to your tables without manual classification, and the AI generates these values as part of the synthetic data generation process.
Can you apply constraints, gender and locale options?
Yes. Constraints can also be applied during generation. For datetime columns, for example, you can define a specific range to ensure the generated values fall within a desired window.
If you want generated names to reflect a particular gender, that can be configured as well. And if you need the synthetic data to reflect details from a specific locale, that option is available too.
Data model configuration at a glance
All of the above configuration is done within the Data Model component of TDM. This can be done across all tables in a single pass before moving to the generation phase.
1. Go to the data model
2. Use AI to auto-detect entities and verify them
3. Load sample data
4. Let AI define target types for synthetic data and verify again
5. Make adjustments as required for enterprise needs
How do you generate and load synthetic data with TDM datasets?
Once configuration is complete, the process moves to the Dataset component of TDM, where you choose the output destination, select tables, generate the data and load it into the target.
Choose where the generated data is saved
Here, you define where you want to save the generated data and how. This includes choosing whether to save the synthetic data to CSV files, load it directly into tables in an existing database, into a specific schema, and so on.
Select tables and run generation
Next, you select the tables for which you want to generate data, with the option to do this for all tables or only selected ones.
Data generation typically takes a few seconds to a few minutes, depending on the volume configured. For 100 or 1,000 rows, generation completes in approximately 5 seconds. When generating across multiple tables, Datagaps processes them in parallel within the same time frame.
Load data into the target table or file
Once data is generated, it can be loaded into the target table or file. If a table is selected as the target, the schemas must be clearly defined to avoid conflicts during insertion.
What happens if sample data is missing?
If sample data was not loaded for some tables during configuration, the tool will fail during generation and prompt you to go back and add sample data. This is by design. Sample data is what enables the system to identify patterns within your data so the output remains relevant to your use case.
When all items are configured correctly, data generation will succeed consistently.
Choose save behavior and confirm the load
When loading into the target table, you can choose the save behavior: append, drop, or truncate. The appropriate choice depends on how you intend to proceed with testing once the data is available.
- If the load is successful, a confirmation is shown. If not, Datagaps provides logs so you can trace and resolve the exact issue. Loading of final synthetic data is straightforward whether you are saving to a file such as CSV or directly to a table. For table targets, validate the schema and the process is complete.
Conclusion
That is the complete process for generating synthetic data at scale for enterprise environments, regardless of the data source, whether Microsoft Fabric or others. Datagaps supports each step of the testing process and provides all the tools required for testing any part of an analytical workflow.
The key differentiator is AI integration at every step. The same work done manually could take weeks to replicate data correctly, and even then, it would be difficult to introduce the same level of diversity, especially for edge cases and unexpected patterns.
With AI-powered generation, teams can produce large volumes of synthetic data, introduce varied patterns and data rules, and run comprehensive testing of pipelines before anything reaches production.
Frequently Asked Questions
How does Datagaps TDM generate synthetic data for Microsoft Fabric?
TDM uses two components. In the Data Model you connect a Fabric lakehouse or warehouse, classify entities, load sample data and set target data types. In the Dataset you choose the volume, select tables and save the output to files such as CSV or load it directly into tables. AI/ML capabilities generate the data from patterns in the sample.
Does Datagaps TDM mask PII in synthetic test data?
Yes. When a column is classified as one of TDM’s defined entities, Datagaps treats it as PII or confidential and masks it during generation. Entities can be set manually or auto-detected with AI, which classifies names, payment details, addresses and date-time values. Datagaps supports GDPR, CCPA and HIPAA-aligned workflows.
Why does TDM need sample data before generating synthetic data?
Sample data lets the AI identify patterns in existing values so generated output stays relevant to your use case. By default TDM takes 5,000 rows from the connected source, which you can increase, decrease or filter with a WHERE clause. If sample data is missing for a table, generation fails by design and prompts you to add it.
How does TDM keep referential integrity across related Fabric tables?
TDM lets you define relationships between tables and map columns to their parent column in another table. A foreign key, for example, must be mapped correctly and must be available in its source table; otherwise the generated data will lack referential integrity. Defining these mappings during data model configuration keeps related tables consistent.
How fast is synthetic data generation with Datagaps TDM?
Generation typically takes a few seconds to a few minutes, depending on the configured volume. For 100 or 1,000 rows it completes in approximately five seconds, and multiple tables are processed in parallel within the same time frame. The full workflow reduces what could take weeks of manual effort to under an hour.
RajMohan Achanta
Associate Product Manager, Datagaps
Associate Product Manager at Datagaps. Shapes the product experience across ETL Validator, BI Validator, and Data Quality Monitor.
Avinash Keshri
Head, Product Marketing
Head of Product Marketing at Datagaps and IIM Bangalore alumnus. 13+ years of experience in commercializing AI and data platforms across global markets.
