Datagaps is the only company to be listed in Gartner® DataOps Tools & Data Observability market guides

Menu Close
Request a Demo
Book Icon ON THIS PAGE
Explore DataOps Suite
  • Can ETL Validator help compare data from multiple sources?
  • Does ETL Validator support Continuous Integration?
  • Is there any way to schedule tests and receive email notification?
  • Is there reporting available for Test Runs?
  • What is File Watcher?
  • What if my data source is not supported by ETL Validator?
  • Is there a free trial available for ETL Validator?
  • What is a repository and workschema? what databases are supported as repository?
  • What are the Architectural components of ETL Validator?
  • What are the System Requirements for doing a pilot?
  • Can ETL Validator help compare data from multiple sources?
  • Does ETL Validator support Continuous Integration?
  • Is there any way to schedule tests and receive email notification?
  • Is there reporting available for Test Runs?
  • What is File Watcher?
  • What if my data source is not supported by ETL Validator?
  • Is there a free trial available for ETL Validator?
  • What is a repository and workschema? what databases are supported as repository?
  • What are the Architectural components of ETL Validator?
  • What are the System Requirements for doing a pilot?

Data Migration Testing: The Complete Guide to Automated Migration Validation

Data migration testing is the process of validating that data moves correctly from a legacy system, on-premises database, or existing cloud environment to a new target platform, with zero data loss and all transformation rules applied correctly. It covers source-to-target validation, reconciliation, and regression testing before, during, and after the migration cutover. Datagaps ETL Validator automates migration validation across Snowflake, Databricks, Azure Synapse, and 200+ source and target systems. 

Key Takeaways

dot

Data migration testing validates that every record moves correctly from source to target — zero data loss, zero transformation errors, zero broken key relationships.

dot

Failed migrations are typically irreversible at scale catching discrepancies before cutover costs a fraction of fixing them in production.

dot

Datagaps ETL Validator validates 100% of records across 200+ systems eliminating the sampling gap that leaves most migration projects blind to edge-case errors.

dot

Agentic AI generates test cases directly from the migration mapping specification eliminating weeks of manual test authoring for large migration programmes.

dot

Datagaps is a certified Snowflake Select Technology Partner and Databricks Technology Partner, and is listed in Gartner's Market Guide for DataOps Tools.

What Is Data Migration Testing?

Data migration testing is the discipline of validating data during a migration project. Unlike ongoing ETL testing, which validates pipelines on a recurring basis, migration testing is project-scoped: there is a defined source system, a target platform, a transformation mapping, and a cutover date. Validation happens across three phases. 

Pre-migration: Profiling the source data to establish a quality baseline and identify anomalies before anything moves.

During migration: In-flight validation to confirm that records, schemas, and transformation rules apply correctly as data moves.

Post-migration: Full reconciliation between source and target to confirm zero data loss and correct transformation at the field level. 

Datagaps ETL Validator automates all three phases, connecting to both the legacy source and the new target simultaneously to validate 100% of records at every stage. For a deeper look at the types and challenges of data migration testing, see the concept guide. Teams using ETL Validator for ongoing pipeline testing can apply the same platform to migration projects, the ETL Testing Guide covers the recurring validation layer that follows every successful migration. 

Why Does Data Migration Testing Matter?

According to Gartner, poor data quality costs organisations an average of $12.9 million per year. In migration projects, the stakes are higher because migration is typically irreversible at scale: once cutover happens, rolling back is exponentially more expensive than catching errors before. The consequences are direct: delayed go-live dates when validation reveals data discrepancies late in the project, corrupted production data in the new platform that propagates to every downstream report and model, and compliance failures when regulated data does not reconcile between source and target. Understanding the cost benefit of cloud migration starts with understanding what happens when migration testing is skipped or under-scoped. The common pitfalls of cloud data migration are almost always testing pitfalls.

Check out the information on Data Migration Testing at a conceptual level here: Data Migration Testing Concept.

What Are the Common Data Migration Testing Challenges?

Data teams report an average of 67 data incidents per month (2026 Wakefield/Monte Carlo study). During migration projects, the incident rate spikes because every record, schema, and transformation is in motion simultaneously. The data validation challenges in cloud migration compound when source systems predate the tools designed to test them. 

Data volume

Enterprise migrations move billions of records. Manual testing or sampling covers a fraction of the dataset, leaving the majority of records unvalidated. The errors that matter most hide in the records nobody checked.

Schema transformation

Source and target schemas rarely match. Every column mapping, data type conversion, and naming convention change needs validation. A single incorrect mapping can corrupt an entire table silently, an Oracle NUMBER(18,4) mapped to a Snowflake DECIMAL(15,2) silently loses two decimal places on every record. 

Legacy system complexity

Mainframes, flat files, proprietary formats, and decades-old database structures do not map cleanly to modern cloud platforms. SAP S/4HANA migrations are a prime example: Material Master data with complex hierarchies and custom fields requires specialised test automation that generic tools cannot handle.

Cutover window pressure

Migration testing must happen within tight downtime windows. If validation takes longer than the window allows, the project either delays or goes live without full testing. Neither option is acceptable.

Reconciliation at scale

Aggregate totals between source and target may match while row-level discrepancies hide in the detail. True reconciliation means validating every record, not just the summary. For a forward look at how the future of automated migration testing is evolving, see the full analysis.

Datagaps ETL Validator addresses each of these through 100% row-level validation, automated schema mapping verification, agentic AI test generation, and reconciliation that runs within cutover windows. Also see ETL Validator for data migration testing for the applied approach.

What Are the Core Migration Validation Checks?

ETL Validator runs nine validation checks during a migration through its DB Flows module: the product term for migration test cases. Each DB Flow executes SQL-based comparison logic between legacy source and new target, using a Workschema (a temporary staging schema hosted inside your own Snowflake, Redshift, Oracle, or PostgreSQL environment) as the comparison engine. Data never leaves your infrastructure boundary.

1. Row Count Validation - Did Every Record Make the Crossing?

What it validates: That every source record arrived at the target nothing dropped during extraction or load.

ETL Validator executes COUNT(*) at table level, partition level (by date range, batch ID, or region), and schema level across all migrated tables. Dynamic Parameters allow the same DB Flow to run against each batch increment without duplicate configuration which is critical for staged migrations.

Exception output:

Table: CUSTOMER_MASTER | Source (Oracle): 14,823,117 | Target (Snowflake): 14,810,043 | Delta: −13,074 (−0.088%) | FAIL | Root cause: ETL filter incorrectly excluded enterprise customers with legacy account codes prefixed “E-“.

2. Data Content Validation - Do Field Values Match?

What it validates: That actual field values in every migrated record match between source and target.

The Query Compare mechanism extracts data from both systems into the Workschema and runs a SQL JOIN-based diff, identifying every row where source and target values diverge. Configurable comparison modes: exact match, threshold-based (±tolerance for numeric rounding between Oracle NUMBER and Snowflake DECIMAL), or column exclusion (audit timestamps, migration-generated surrogate keys).

Exception output:

CUSTOMER_ID = 8821943 | Column: ANNUAL_REVENUE_USD | Source: 14,750,000 | Target: 1,475,000 | Variance: −90% | Root cause: decimal shift in Oracle NUMBER(18,3) to Snowflake DECIMAL(15,0) conversion — scale precision dropped.

3. Data Type, Format and Schema Validation

What it validates: That column names, data types, precision, nullable constraints, date formats, and character encoding match between source and target schemas.

ETL Validator's Metadata Compare module checks the structural layer independently of data content like, catching data type mismatches, precision/scale loss, nullable constraint changes, column presence, and character encoding issues. Particularly critical when migrating from Oracle, SAP, or IBM DB2 to cloud platforms where type systems differ significantly.

Exception output:

Column: EFFECTIVE_DATE | Source (SAP): VARCHAR(8) YYYYMMDD | Target (Databricks): DATE | Issue: 127 records with SAP sentinel value “00000000” failed DATE conversion. Expected: NULL in target.

4. Transformation Logic Validation - Did Business Rules Apply Correctly?

What it validates: That transformation logic: concatenation, lookups, conditional logic, currency conversion, status remapping applied correctly to every record.

The expected transformation is expressed as SQL against the source and compared against the transformed output in the target. This gives row-level proof that business rules executed correctly on every record - not just a sample.

Exception output:

Transformation: TRIM(FIRST_NAME)||” “||TRIM(LAST_NAME) | Row: CUST_ID = 442891 | Source derived: “Robert Smith” | Target actual: “Robert  Smith” | Root cause: TRIM not applied to LAST_NAME. Affects 8,243 records.

Key takeaway:

Unlike schema tools that check structure, transformation validation gives row-level proof that business logic applied correctly to every record — not just an aggregate check that the totals look right.

5. Referential Integrity Validation - Did Key Relationships Survive?

What it validates: That primary key, foreign key, and parent-child relationships between tables survived the migration intact.

The Foreign Key Rule validates that every FK value in a child table has a matching PK in the parent table. Referential integrity failures produce orphaned records that cause incorrect aggregations in every BI report joining the affected tables silently, with no error visible to the business user.

Exception output:

FK violation: ORDERS.CUSTOMER_ID → CUSTOMERS.CUSTOMER_ID | Orphaned records: 2,847 | Root cause: CUSTOMERS migration filtered inactive accounts, but ORDERS retained all orders including those referencing inactive customers.

6. Completeness and Null Handling - Were Any Records Dropped?

What it validates: No expected records are absent from the target; no column has unexpected NULL values in fields that should contain data.

ETL Validator checks completeness at two levels: full-set record comparison (every source record must have a target match), and column-level null validation via Attribute Rule (non-nullable columns flagged when NULLs appear in target). Data Profile comparison catches statistical null rate shifts, a column with 2% nulls in source arriving with 15% nulls in target is flagged even without an explicit rule.

Exception output:

Table: PRODUCT_INVENTORY | Source: 892,341 | Target: 889,102 | Missing: 3,239 records | Root cause: ETL condition treated “OBSOLETE” and “DISCONTINUED” as equivalent — 3,239 records excluded from migration scope.

7. Aggregation and Reconciliation Validation

What it validates: Summary totals, financial rollups, and calculated aggregations match between source and target after migration.

ETL Validator runs GROUP BY aggregations against source detail and compares against target summary values. Multiple Data Compare validates cross-source aggregations — for consolidated migrations where multiple legacy source tables load into one target, it confirms the consolidated total equals the sum of all contributing sources.

Exception output:

SUM(ORDER_AMOUNT) by [REGION=APAC, FY=2024] | Source: $2,847,920,315 | Target: $2,839,441,220 | Delta: −$8.5M (−0.30%) | Root cause: Oracle NUMBER to Snowflake DECIMAL rounding at row level across 36.9M records.

8. Duplicate Detection - Were Any Records Loaded Twice?

What it validates: Records in the target do not violate uniqueness constraints a common failure when a migration batch retries after a connection timeout without idempotency checks.

The Duplicate Check rule runs a GROUP BY with HAVING COUNT(*) > 1 against configured key columns. Duplicate records in dimension tables cause downstream BI reports to multiply fact aggregations through inadvertent many-to-many joins double-counting revenue across every report that joins to the affected dimension.

Exception output:

Key: CUSTOMER_MASTER.CUSTOMER_ID | Duplicates: 1,847 rows | Root cause: migration batch ran twice on 50K-record subset during retry after Snowflake connection timeout — idempotency not enforced.

9. SCD Type 2 Validation - Did the Full Dimension History Survive?

What it validates: The complete history of dimension changes survived the migration not just current-state records.

ETL Validator has documented support for SCD Type 2 validation checking three dimensions: historicisation (new row inserted with correct EFF_FROM_DATE; prior row EFF_TO_DATE set correctly), attribute accuracy (new row carries updated source attributes; historical row retains originals), and current row integrity (exactly one CURRENT_FLAG = TRUE per dimension member).

Key takeaway:
For data warehouse migrations, SCD Type 2 validation confirms that the full historical record of dimension changes survived intact — not just current-state records. This is the check that separates a complete migration from a partial one. 

See all 9 checks in action. Start a 14-day free trial → Run your first migration validation in under an hour.

What Are Data Migration Testing Best Practices?

In summary: six practices that separate successful migration projects from failed ones.

Step 1

Run pre-migration data profiling before anything moves. Profile the source system to establish a quality baseline null rates, cardinality, distribution, edge cases. If you do not know what the source looks like before migration, you cannot confirm the target matches it after. ETL Validator's JDBC-Profile captures this baseline.

Step 2

Map and verify every source-to-target column before running the first test. Migration validation starts with the mapping spec. If the mapping is wrong, every test built on it validates the wrong thing. Verify column mappings, data type conversions, and naming conventions independently before running full validation.

Step 3

Run validation during the migration window, not after cutover. Catching errors during migration gives you time to fix and re-run. Catching them after cutover means the damage is already in production. Design your testing to fit within the Cutover Window. not around it.

Step 4

Validate at every transformation layer, not just source vs. final target. Migrations with staging tables, intermediate transforms, or medallion architectures (Bronze/Silver/Gold) need validation at each layer. An error in the Bronze layer compounds silently through every downstream layer.

Step 5

Automate reconciliation for the full dataset. Manual spot-checking or aggregate-level comparison hides row-level discrepancies. Automated AI reconciliation validates every record, categorises discrepancies by type and business impact, and flags the exact rows where source and target do not match.

Step 6

Define rollback criteria before go-live, not after. Set explicit thresholds: how many discrepancies trigger a rollback? What severity level blocks cutover? Build these into the validation reporting so the go/no-go decision is data-driven, not a judgment call under pressure.

For the complete migration testing methodology, download the ETL Testing Playbook. 

Download: The Cost Benefit of Data Migration to the Cloud →  The business case framework for cloud migration investment.

How Do You Validate Data Migration to Snowflake?

Datagaps is a certified Snowflake Select Technology Partner the integration is production-verified and validated through the Snowflake partner programme. Enterprise Snowflake customers who evaluate tools through the Partner ecosystem can confirm this credential directly. 

Snowflake is the most common cloud migration target for enterprise data teams, and it introduces platform-specific validation requirements that generic testing approaches miss. ETL Validator connects natively to Snowflake as both a migration source and target with three authentication methods: OAuth (for Azure AD/Okta SSO environments), Key Pair Authentication (RSA-based, no password transmitted preferred for enterprise service accounts), and User Profile Authentication. Credentials are stored in Azure Key Vault or AWS Secret Manager, never in plain text. 

Workschema Inside Snowflake - Data Never Leaves Your Account

The most important architecture detail: ETL Validator’s Workschema is created inside the Snowflake account itself. Source data from Oracle, SAP, or IBM DB2 is extracted into the Workschema and the comparison SQL runs as native Snowflake SQL using the Snowflake virtual warehouse’s multi-cluster compute. Data never leaves your Snowflake account. For migrations with data residency requirements, validation is compliant by design. 

Snowflake-Specific Migration Validation Capabilities

Stages:

ETL Validator validates flat files in Snowflake stages (Parquet, AVRO in S3 or ADLS) via its file connectors before COPY INTO executes catching file-level issues before they reach the table layer. Especially valuable for mainframe-to-Snowflake migrations using COBOL copybook file extracts.

Semi-structured data (VARIANT):

Snowflake's VARIANT column type stores JSON, AVRO, and XML natively. ETL Validator validates VARIANT columns using Snowflake's colon notation (src:field.nested_field) or GET_PATH validating individual nested fields within VARIANT columns rather than treating the entire column as opaque.

Time Travel:

AT/BEFORE clauses in source SQL compare the current target state against a historical source snapshot solving the race condition where the source keeps changing during the migration window. Record the extraction timestamp; after migration completes, ETL Validator queries Snowflake AT that exact timestamp for a clean point-in-time comparison.

Zero-Copy Clones:

Snowflake clones create an instantaneous schema copy without duplicating data. ETL Validator validates against the clone rather than the live target eliminating false mismatches from post-cutover writes landing during the validation run.

For teams migrating from Oracle, SAP, or mainframe systems, see data validation in Snowflake migration and the complete guide to Snowflake data migration testing and validation.

How Do You Validate Data Migration to Databricks?

Datagaps is a certified Databricks Technology Partner validated integration for Delta Lake, Unity Catalog, and medallion architecture migration validation. 

ETL Validator connects to Databricks with built-in cluster management: it starts, stops, and configures clusters for validation execution, with job clusters (cost-optimised, on-demand) or interactive clusters (always-available). For large-scale lakehouse migrations, the DataOps Engine runs on Databricks as a distributed Spark execution environment handling billion-row migrations using native parallelism.

Delta Lake tables are queried directly through the JDBC connector Delta’s ACID guarantees ensure ETL Validator reads consistent data, not a partially-written intermediate migration state. Delta Time Travel (VERSION AS OF and TIMESTAMP AS OF) works in ETL Validator source queries for phase-boundary validation, compare target state after each medallion layer against a specific Delta table version. 

For teams migrating to Databricks from legacy data warehouses, the lakehouse architecture introduces multiple transformation stages. ETL Validator validates at each medallion layer, i.e., Bronze (raw ingestion completeness), Silver (cleansing, type casting, deduplication), Gold (aggregations, derived metrics, dimensional joins) as separate DB Flows in a dependency Pipeline so Bronze errors do not propagate silently into Gold. Unity Catalog integration tracks lineage from Gold back to the Bronze source. For full technical depth, download the Databricks Lakehouse Migration & Data Validation whitepaper.

How Do You Validate Data Migration to Azure?

ETL Validator connects to Azure Synapse Analytics and Azure SQL for migration validation across the full Microsoft data stack supporting both dedicated SQL pools (MPP architecture, Workschema hosted inside Synapse) and serverless SQL pools (querying ADLS data directly, useful for validating the data lake layer before processing into the dedicated pool). 

For Oracle customers migrating to Azure, ETL Validator validates both the Oracle to Azure Synapse migration path and the Oracle to Azure SQL Server migration path, covering schema mapping, data content, transformations, and full reconciliation for each route. Credentials are stored in Azure Key Vault native integration, no plain-text credential storage.

Microsoft Fabric - The 2026 Microsoft Migration Target

Microsoft Fabric is the unified Microsoft analytics platform integrating Azure Synapse, Power BI, and Azure Data Factory into a single SaaS environment. ETL Validator’s dedicated Microsoft Fabric connector (OAuth authentication) supports migration validation across Fabric semantic models and lakehouses. Fabric’s OneLake storage layer uses Delta Parquet format for all workloads the same format as Databricks Delta Lake, meaning the same Delta validation patterns (Time Travel, ACID consistency) apply to Fabric migrations. 

Organisations migrating from Azure Synapse to Fabric can use existing ETL Validator test suites against the Fabric data layer with minimal reconfiguration. Azure Synapse Pipelines integrate with Azure DevOps for CI/CD after each migration batch completes, the DataOps CLI triggers ETL Validator automatically, with pass/fail status gating batch promotion. 

Validate your cloud migration. Start a 14-day free trial → with native Snowflake, Databricks, Azure Synapse, and Microsoft Fabric support.

How Does ETL Validator Automates Data Migration Testing?

ETL Validator follows a four-step migration validation lifecycle. The architectural differentiator is the Workschema: comparison SQL runs natively inside your own database environment (Snowflake, Redshift, Synapse, or Oracle) using your compute, not an application layer. Data never leaves your infrastructure. For migrations with data residency requirements, validation is compliant by design. 

Step 1: Connect - Legacy Source AND Modern Target, Simultaneously

ETL Validator maintains two active connections throughout the migration window: one to the legacy source (the ground truth), one to the new target (the migrated data). No-code configuration — no custom scripting required.

Legacy source connections

Mainframe: IBM DB2 for z/OS, IBM AS/400 (iSeries), IBM DB2 for LUW; COBOL copybook format files from mainframe extracts.

Oracle: Oracle on-premises (with Wallet auth for Autonomous Database), Oracle Fusion/ERP Cloud, Oracle E-Business Suite.

SAP: SAP HANA supporting S/4HANA, BW, and ECC migrations.

Legacy analytics: IBM Netezza (PDA), IBM Informix, Pivotal Greenplum common in FSI and insurance environments.

Microsoft on-premises: SQL Server (all versions), Hadoop Hive 2, Apache Impala.

Flat files: CSV, Fixed Length, COBOL copybook, Excel, JSON, AVRO, Parquet, XML from SFTP, ADLS, S3, GCS, SharePoint.

Modern target connections

Snowflake | Azure Databricks | Azure Synapse | Microsoft Fabric | Amazon Redshift | Google BigQuery | PostgreSQL | MySQL | SAP HANA via pre-built JDBC connectors. Credentials stored in Azure Key Vault, AWS Secret Manager, or HashiCorp Vault not in plain text.

Step 2: Generate - AI Test Cases from the Migration Mapping Spec

Mapping specification import: The Mapping Manager imports the Excel source-to-target mapping document directly. ETL Validator reads source-to-target column mappings and generates DB Flows automatically covering row count, data content, type validation, and transformation checks. A 100-table migration mapping document produces a complete, deployable test suite without manual test authoring.

For columns that do not resolve automatically, AI-Powered Mapping suggests matches by name similarity and data type compatibility. For migrations without a formal mapping document, Generate DB Flows reads source and target schema metadata and auto-generates a full test suite from structure alone, a 50-table migration: In hours, not weeks.

Setup time: Legacy JDBC connection (Oracle/SAP): 20–30 minutes. Cloud target connection (Snowflake/Databricks): 10–15 minutes. Mapping spec import and test generation for a 100-table migration: 30–60 minutes. First validation run against both source and target: hours from project start.

Step 3: Execute 100% Row Validation Within the Cutover Window

Three mechanisms make cutover-window validation possible:

Workschema native execution: Comparison SQL runs inside your target database using its own compute (Snowflake multi-cluster warehouse, Synapse MPP, Databricks cluster). ETL Validator orchestrates the SQL it does not pull rows into an application layer. This is what makes billion-row validation possible within a cutover window.

Parallel Pipeline execution: 200 tables validate concurrently with configurable parallelism and dependency ordering, dimension tables validate before fact tables, preventing FK cascade failures from confusing root cause analysis.

Incremental batch validation: Dynamic Parameters limit each run to the current batch (by batch ID, date partition, or watermark). At cutover, a final reconciliation confirms full source-to-target match across all batches.

Scale-out: For very large migrations, the DataOps Engine scales horizontally on Azure Databricks clusters, Apache YARN, Amazon EMR, or Kubernetes (AKS, EKS, GKE) executing validation as distributed Spark jobs. Datagaps customers have validated 500B+ records through this architecture.

CI/CD integration: Azure DevOps Pipelines, GitHub Actions, and Jenkins via the DataOps CLI trigger validation automatically after each migration batch load pass/fail exit codes gate batch promotion. See AI-powered reconciliation for large-scale migrations for the intelligent layer on top of standard validation.

Step 4: Report Drill-Down Exception Output and Audit Trail

Every execution produces a structured, drillable exception report:

Pipeline summary: Total tables validated, pass/fail/warning per table, total records matched vs. unmatched, execution time (confirming the run fits within the cutover window), and go/no-go threshold assessment.

Row-level drill-down: Join key (row identifier), every column that diverged, source value, target value, calculated variance. Column-level failure summary distinguishes systematic mapping errors (80% of rows) from isolated data anomalies (3 specific rows).

AI reconciliation: Categorizes discrepancies as data error, transformation error, mapping error, or system error prioritising by business impact so the remediation team addresses revenue-affecting discrepancies first.

Audit trail: Full run history: who triggered validation, when, against which systems, with what parameters, and row-level results serving as compliance artifacts for regulated migrations.

Export and notifications: Results export to Excel, ADLS, or S3. Email, Microsoft Teams, and Slack notifications on completion.

Why Is Datagaps the Complete Platform for Data Migration Testing?

Migration testing does not end when data lands in the new platform. Reports built on the migrated data need re-validating. Data quality needs monitoring post-cutover to catch issues in the first weeks of production. Source-to-target reconciliation needs to run continuously, not just during the migration window. And if the migration includes a BI platform change such as moving from Tableau to Power BI the reporting layer needs its own validation pass.

Datagaps covers the full migration testing lifecycle through a suite of integrated products:

ETL Validator

Pipeline and migration validation: 100% row-level source-to-target validation throughout the migration project.

BI Validator

Post-migration report and dashboard testing: confirms that BI reports built on the migrated data display correct values after cutover.

Data Quality Monitor

Continuous data quality scoring after cutover: monitors the new platform for data drift, anomalies, and freshness issues in the weeks after go-live.

Data Reconciliation

Ongoing source-to-target matching: continuous reconciliation between legacy source and new target during any parallel-run period.

DataOps Suite

All four products in a single platform with shared pipelines, scheduling, CI/CD integration, and a unified audit trail.

Datagaps is advancing migration validation from agent-assisted automation to a fully AI-native workflow. The AI Workflow ingests the migration mapping between source and target, automatically builds lineage across the full migration path, and auto-generates validation test cases covering every transformation stage, not just the final source-to-target comparison. For complex migrations involving hundreds of tables across multiple systems, this eliminates weeks of manual test case authoring. The platform’s ETL-scale processing capability, built on Spark with native support for billions of records, means validation runs within cutover windows regardless of data volume. 

Talk to a migration testing expert. Request a migration validation demo →  See the full lifecycle in action. 

Datagaps ETL Validator: Solution Brief

Datagaps ETL Validator is the automated data migration testing platform for teams that need every record validated before, during, and after cutover, with zero tolerance for data loss. It validates 100% of records at the row level across 200+ source and target connectors using Datagaps US-patented Extract-Load-Validate architecture not sampling, not spot-checking, not rule-based monitoring. 
Setup is fast: connect your legacy source and new target using pre-built connectors, import the migration mapping specification and let agentic AI generate test cases automatically, and run. Validation completes within cutover windows, producing exception reports with drill-down to the exact record and field where source and target do not match. ETL Validator is available standalone or as part of the Datagaps DataOps Suite. 

Enterprise teams use ETL Validator to validate cloud migrations, automate reconciliation, and build audit-ready validation into their migration governance process: 

Cloud migration:

60% reduction in overall deployment time, 70% faster validation of new datasets, 95% improvement in overall data quality.

Fortune 100 Financial Services (mainframe to Snowflake):

100% automated test coverage, up to 35% faster migration delivery, 45–60% less testing effort.

CPG leader (Oracle to Snowflake):

Up to 60% reduction in migration testing time, 70% cost savings in QA, 95% reduction in data quality testing effort.

French consumer brand:

45–60% reduction in migration testing time, 30% reduction in total cost of ownership, 100% automated testing coverage.

Datagaps is recognised in Gartner’s Market Guide for DataOps Tools. Certified Snowflake Select Technology Partner and Databricks Technology Partner.

Start Free Trial
Download the ETL Testing Playbook

How Does ETL Validator Compare Against Other Migration Testing Approaches?

When evaluating migration testing approaches, three criteria matter most: Record coverage (can it validate 100% of migrated data?), Cutover-window performance (can it run within your downtime window?), and Reusability (can you use the same test plans for the next migration?).

CriteriaDatagaps ETL ValidatorQuerySurgeiceDQDIY/LLM-Built
Record coverage100% row-levelQuery-based samplingRule-based monitoringDepends on script quality
Setup for migrationNo-code, connect source + targetTool-specific configurationRule configurationPrompt + debug per migration
Schema mapping validationAutomaticPartialPartialNo persistent schema awareness
ReconciliationBuilt-in source-to-targetQuery-basedLimitedManual per run
Cutover-window testingAutomated, runs in minutesVariableVariableUntested under time pressure
Audit trailFull exception reportsPartialPartialNone
Reusability across migrationsTest plans reusablePartially reusablePartially reusableRe-prompt and re-debug each time
ETL-to-BI pipeline coverageFull (with BI Validator)ETL layer onlyETL layer onlyETL layer only
Validation architectureUS-patented Extract-Load-ValidateProprietaryProprietaryNot applicable

The fourth column is worth emphasizing. With ChatGPT, Copilot, and other AI coding assistants, it is tempting to prompt an LLM to generate migration validation scripts. It works for a proof of concept, but at enterprise scale, LLM-generated scripts have no persistent schema awareness, no reconciliation capability, no audit trail, and no way to run within a cutover window under time pressure. Every migration means re-prompting, re-testing, and re-debugging. ETL Validator handles all of this as a maintained platform with 200+ connectors that your team doesn’t have to support.

Data Migration Testing in Practice: Who Uses ETL Validator?

Cloud Migration

An enterprise team migrating production applications to the cloud used ETL Validator to ensure zero data loss across the full migration. Result: 60% reduction in overall deployment time, 70% faster dataset validation, 95% improvement in data quality.  Migrating Production Applications to the Cloud →

Financial Services - Mainframe to Snowflake

A Fortune 100 financial services company migrated mission-critical mainframe data to Snowflake with zero tolerance for data loss. Result: 100% automated test coverage, up to 35% faster migration delivery, 45–60% less testing effort. → Fortune 100 Firm Automates Snowflake Data Pipeline Validation →

Consumer Packaged Goods - Oracle to Snowflake

A CPG leader migrating from an Oracle legacy system to Snowflake validated complex product, supply chain, and sales pipelines. Result: 60% reduction in migration testing time, 70% QA cost savings, 95% reduction in data quality testing effort. → Oracle to Snowflake ETL Validation →

Consumer Brand - Migration Time Reduction

A French consumer brand compressed multi-week manual testing cycles into days. Result: 45–60% reduction in migration testing time, 30% reduction in total cost of ownership, 100% automated testing coverage. → ETL Automation and Validation Process →

Resources

Whitepaper

The Cost Benefit of Data Migration to the Cloud

Whitepaper

Databricks Lakehouse Migration & Data Validation

eBook

The ETL Testing Playbook

Further Reading

Migration Methodology

Migration Platform Specifics

Cross-Pillar Links

ETL Testing: The Complete Guide →

ongoing pipeline validation with ETL Validator after migration is complete.

BI Testing: The Complete Guide →

Validate reports and dashboards on the new platform post-migration

Frequently Asked Questions

What is Data Migration testing?

Data migration testing is the process of validating that data moves correctly from a source system to a target platform during a migration project. It covers row counts, data content, schema mapping, transformation logic, referential integrity, and reconciliation between source and target before, during, and after cutover. 

What is the difference between data migration testing and ETL testing?

Data migration testing is project-based: it validates a specific migration event with a defined source, target, and cutover date. ETL testing is ongoing: it validates pipelines that run on a recurring schedule. The validation checks are similar, but the scope, timeline, and risk profile are different. Migration testing has a hard deadline; ETL testing is continuous. 

How long does automated migration testing take to set up?

With ETL Validator, initial setup typically takes hours, not weeks. Legacy JDBC connections (Oracle, SAP HANA) take 20–30 minutes. Cloud target connections (Snowflake, Databricks) take 10–15 minutes. Agentic AI generates test cases from the migration mapping specification automatically. 

Can ETL Validator validate migration to Snowflake?

Yes. ETL Validator is a certified Snowflake Select Technology Partner. It connects natively to Snowflake and validates data across stages, semi-structured VARIANT data, and full source-to-Snowflake reconciliation including Time Travel snapshots and zero-copy clones for race-condition-free validation. 

Can ETL Validator validate migration to Databricks?

Yes. ETL Validator is a Databricks Technology Partner. It validates data across the medallion architecture (Bronze, Silver, Gold layers), tests Delta Lake table integrity with Time Travel, and supports Unity Catalog integration for governed migration validation. 

What about Azure Synapse, Microsoft Fabric, and Azure SQL migration?

Yes. ETL Validator connects to Azure Synapse (dedicated and serverless SQL pools), Azure SQL Server, and Microsoft Fabric (OAuth connector, Delta Parquet OneLake). It supports Oracle-to-Azure Synapse and Oracle-to-Azure SQL migration paths with full schema mapping and reconciliation. 

How does ETL Validator handle billions of rows during migration?

ETL Validator validates 100% of records using the US-patented Extract-Load-Validate architecture. Comparison SQL runs natively inside your database — Snowflake, Redshift, or Synapse — using their own compute, not an application layer. For very large migrations, the DataOps Engine scales horizontally on Databricks clusters, Apache YARN, or Kubernetes. 

Can I use ChatGPT or an LLM to build migration testing instead?

For a one-off validation script, yes. For enterprise-scale migration testing, no. LLM-generated scripts lack persistent schema awareness, have no mapping spec import capability, produce no audit trail, and have never been tested under cutover-window time pressure. Every migration means re-prompting and re-debugging. ETL Validator handles all of this as a maintained platform with 200+ connectors. 

What is data reconciliation in the context of migration testing?

Data reconciliation during migration compares source and target data to confirm they match after migration going beyond row counts to validate at the field level and catching discrepancies that aggregate checks miss. ETL Validator includes AI-powered reconciliation with drill-down exception reporting and automatic categorisation of discrepancies by type and business impact. 

Does ETL Validator provide an audit trail for migration validation?

Yes. Every validation run produces a full exception report with timestamps, record-level results, and drill-down to specific discrepancies. Run history captures who triggered validation, when, against which systems, and with what parameters. These serve as compliance artefacts for regulated migrations and provide the evidence needed for go/no-go cutover decisions. 

×