<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
>
<channel>
<title>Snowflake Archives - Datagaps | Gen AI-Powered Automated Cloud Data Testing</title>
<atom:link href="https://www.datagaps.com/blog/category/snowflake/feed/" rel="self" type="application/rss+xml" />
<link></link>
<description></description>
<lastBuildDate>Thu, 14 May 2026 07:50:16 +0000</lastBuildDate>
<language>en-US</language>
<sy:updatePeriod>
hourly </sy:updatePeriod>
<sy:updateFrequency>
1 </sy:updateFrequency>
<generator>https://wordpress.org/?v=6.9.4</generator>
<image>
<url>https://www.datagaps.com/wp-content/uploads/cropped-datagaps-favicon-32x32-1-1-32x32.png</url>
<title>Snowflake Archives - Datagaps | Gen AI-Powered Automated Cloud Data Testing</title>
<link></link>
<width>32</width>
<height>32</height>
</image>
<item>
<title>Top 3 ETL Testing Tools: How to Choose the Best Tool</title>
<link>https://www.datagaps.com/blog/top-3-etl-testing-tools/</link>
<dc:creator><![CDATA[Raj Mohan Achanta]]></dc:creator>
<pubDate>Thu, 30 Apr 2026 19:05:05 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Databricks]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7034</guid>
<description><![CDATA[<p>ETL Testing refers to the testing, validation, and analysis of the Extraction, Transformation, and Loading Processes that are part of ETL and ELT Pipelines. As ETL testing refers to “Data-in-Motion” Testing, the unit test architecture and principles slightly differ from “Data-at-Rest” Testing (Warehouse/DB Validation).</p>
<p>The post <a href="https://www.datagaps.com/blog/top-3-etl-testing-tools/">Top 3 ETL Testing Tools: How to Choose the Best Tool</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7034" class="elementor elementor-7034" data-elementor-post-type="post">
<div class="elementor-element elementor-element-3aba7f5 e-flex e-con-boxed e-con e-parent" data-id="3aba7f5" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-0ac44cd elementor-widget elementor-widget-heading" data-id="0ac44cd" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">What are ETL Testing Tools?</h2> </div>
</div>
<div class="elementor-element elementor-element-67925ea elementor-widget elementor-widget-text-editor" data-id="67925ea" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="text-decoration: underline;"><span style="color: #0000ff; text-decoration: underline;"><a style="color: #0000ff; text-decoration: underline;" href="https://www.datagaps.com/data-validation-etl-testing-tools/" target="_blank" rel="noopener"><span style="color: #1967d2; text-decoration: underline;">ETL testing tools</span></a></span></span> are purpose-built platforms that validate data as it moves through extract, transform, and load pipelines. As data pipelines become more complex, organizations rely on ETL testing tools to verify transformations, detect data issues, and maintain trust in analytics.</p><p>While many teams explore general ETL tools, it is important to distinguish between ETL tools used for data movement and ETL testing tools used for validation and quality assurance.</p><p>Looking for a structured starting point? Check out our <span style="text-decoration: underline;"><span style="color: #1967d2;"><a class="underline underline underline-offset-2 decoration-1 decoration-current/40 hover:decoration-current focus:decoration-current" style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/blog/how-to-validate-etl-testing-checklist/" target="_blank" rel="noopener">ETL Testing Checklist</a></span></span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-1e2d7c3 e-flex e-con-boxed e-con e-parent" data-id="1e2d7c3" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-a1a688d elementor-widget elementor-widget-heading" data-id="a1a688d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">When are ETL Testing Tools Used?</h2> </div>
</div>
<div class="elementor-element elementor-element-5963195 elementor-widget elementor-widget-text-editor" data-id="5963195" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL testing tools are primarily used across two major categories of projects where data accuracy is critical:</p> </div>
</div>
<div class="elementor-element elementor-element-04b363b elementor-widget elementor-widget-icon-box" data-id="04b363b" data-element_type="widget" data-e-type="widget" data-widget_type="icon-box.default">
<div class="elementor-widget-container">
<div class="elementor-icon-box-wrapper">
<div class="elementor-icon-box-content">
<h3 class="elementor-icon-box-title">
<span >
1. Data Migration Projects </span>
</h3>
<p class="elementor-icon-box-description">
These involve moving data across systems while ensuring consistency and completeness. Common scenarios include: </p>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-81b768d elementor-widget elementor-widget-text-editor" data-id="81b768d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<ul><li>Application migrations</li><li>Cloud migrations such as moving to <span style="text-decoration: underline;"><span style="color: #1967d2; text-decoration: underline;"><a style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/snowflake-testing-automation/" target="_blank" rel="noopener">Snowflake</a></span></span> or <span style="text-decoration: underline;"><span style="color: #1967d2;"><a style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/databricks-testing-automation/" target="_blank" rel="noopener">Databricks</a></span></span></li><li>Data warehouse migrations such as Teradata to Redshift or Teradata to Databricks</li></ul> </div>
</div>
<div class="elementor-element elementor-element-1dc2108 elementor-widget elementor-widget-text-editor" data-id="1dc2108" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In these cases, ETL testing tools and data testing tools are essential for validating large-scale data movement and ensuring no data loss or transformation errors.</p><p>Need help with data migration? Explore our <span style="text-decoration: underline;"><span style="color: #1967d2; text-decoration: underline;"><a class="underline underline underline-offset-2 decoration-1 decoration-current/40 hover:decoration-current focus:decoration-current" style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/data-migration-testing-automation/" target="_blank" rel="noopener">Data Migration Solution page</a>.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-72c7e93 elementor-widget elementor-widget-icon-box" data-id="72c7e93" data-element_type="widget" data-e-type="widget" data-widget_type="icon-box.default">
<div class="elementor-widget-container">
<div class="elementor-icon-box-wrapper">
<div class="elementor-icon-box-content">
<h3 class="elementor-icon-box-title">
<span >
2. Data Pipeline Testing </span>
</h3>
<p class="elementor-icon-box-description">
These focus on ongoing validation of data pipelines in production environments. Key use cases include: </p>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-9cf7ea7 elementor-widget elementor-widget-text-editor" data-id="9cf7ea7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<ul><li>Verifying data transformations across pipelines</li><li>Ensuring consistency between source and target systems</li><li>Detecting data quality issues early</li><li>Supporting continuous validation as pipelines scale Here, ETL automation testing tools help teams scale validation, reduce manual effort, and maintain data quality across evolving pipelines.<p>Read more on <span style="text-decoration: underline; color: #1967d2;"><a class="underline underline underline-offset-2 decoration-1 decoration-current/40 hover:decoration-current focus:decoration-current" style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/data-testing-concepts/etl-testing/" target="_blank" rel="noopener">ETL Testing</a></span> for data pipeline environments.</p></li></ul> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-b82f1a5 e-flex e-con-boxed e-con e-parent" data-id="b82f1a5" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-8f190b0 elementor-widget elementor-widget-heading" data-id="8f190b0" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Evaluation Criteria: How We Selected and Assessed ETL Testing Tools?</h2> </div>
</div>
<div class="elementor-element elementor-element-e722fe9 elementor-widget elementor-widget-text-editor" data-id="e722fe9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="font-claude-response-body">Modern ETL testing tools are expected to deliver multi-source validation, transformation testing, automation, AI-assisted test creation, and scalability across large data environments. These capabilities formed the basis of our evaluation.</p> </div>
</div>
<div class="elementor-element elementor-element-71ad81b elementor-widget elementor-widget-text-editor" data-id="71ad81b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="font-claude-response-body">Several tools come up frequently in this space. iceDQ, Tosca DI, and Informatica DVO were considered but excluded for specific reasons:</p> </div>
</div>
<div class="elementor-element elementor-element-83f3abe elementor-widget elementor-widget-text-editor" data-id="83f3abe" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>iceDQ:</strong> The on-premise version of iceDQ lacks several core ETL testing capabilities that enterprise teams typically require. The SaaS version is more feature-complete but not suited for teams that need on-premise deployment.</p><p><strong>Informatica DVO:</strong> Informatica DVO is not a standalone ETL testing tool. It runs only within the Informatica platform, making it irrelevant for teams outside that ecosystem.</p><p><strong>Tosca DI:</strong> While Tosca is a popular choice for application and UI testing, Tosca DI is found to be limited in scope for ETL testing and end-to-end pipeline validation, making it a less suitable option for teams with comprehensive data pipeline testing requirements.</p> </div>
</div>
<div class="elementor-element elementor-element-dd5a526 elementor-widget elementor-widget-text-editor" data-id="dd5a526" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="font-claude-response-body">ETL testing tools broadly fall into three categories: purpose-built ETL testing platforms, query-based validation tools, and developer-first testing frameworks. This comparison selects one representative from each category to highlight how different approaches address the same validation challenges. In this comparison, Datagaps ETL Validator represents the purpose-built category, QuerySurge the query-based approach, and dbt Tests the developer-first framework.</p> </div>
</div>
<div class="elementor-element elementor-element-233aa11 elementor-widget elementor-widget-text-editor" data-id="233aa11" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="font-claude-response-body">Evaluation was based on nine criteria that reflect real production requirements: core ETL testing capabilities, automation and CI/CD integration, usability and test authoring, data quality and observability, data contracts and governance, testing scope and coverage, enterprise readiness, scalability and performance, and pricing and accessibility.</p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-a776b00 e-flex e-con-boxed e-con e-parent" data-id="a776b00" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-45a2ac4 elementor-widget elementor-widget-heading" data-id="45a2ac4" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Top 3 ETL Testing Tools: Detailed Comparison</h2> </div>
</div>
<div class="elementor-element elementor-element-8243d2f elementor-widget elementor-widget-text-editor" data-id="8243d2f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Below is a detailed comparison of three widely considered options: <span style="text-decoration: underline;"><span style="color: #1967d2;"><a style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/data-validation-etl-testing-tools/" target="_blank" rel="noopener">Datagaps ETL Validator</a></span></span>, QuerySurge, and dbt tests.</p> </div>
</div>
<div class="elementor-element elementor-element-2b8d724 elementor-widget elementor-widget-html" data-id="2b8d724" data-element_type="widget" data-e-type="widget" data-widget_type="html.default">
<div class="elementor-widget-container">
<!-- ============================================================
TOP 3 ETL TESTING TOOLS: DETAILED COMPARISON
Elementor Custom HTML Block
Desktop: Full-width table without horizontal scroll
Tablet/Mobile: Horizontal scroll enabled
Text Color: #17253D
Font Family: Inter
============================================================ -->
<style>
@import url('https://fonts.googleapis.com/css2?family=inter:wght@300;400;500;600;700&display=swap');
.etl-section {
--font-family: "inter", sans-serif;
--font-size-base: 18px;
--font-weight-normal: 400;
--color-text: #17253D;
--color-accent: #ffffff;
--color-accent-light: #ffffff;
--color-border: #dde5ed;
--color-bg-header: #07152D;
--color-bg-subheader: #356A9B;
--color-bg-alt: #ffffff;
--color-bg-white: #ffffff;
--color-star: #f5a623;
--color-check: #2ecc71;
--color-partial: #f39c12;
--color-cross: #e74c3c;
--border-radius: 8px;
--table-border: 1px solid var(--color-border);
font-family: var(--font-family);
font-size: var(--font-size-base);
font-weight: var(--font-weight-normal);
color: var(--color-text);
line-height: 1.6;
width: 100%;
max-width: 100%;
margin: 0 auto;
padding: 0;
box-sizing: border-box;
}
.etl-section *,
.etl-section *::before,
.etl-section *::after {
box-sizing: border-box;
}
/* ===== Legend ===== */
.etl-legend {
display: flex;
flex-wrap: wrap;
gap: 18px;
margin-bottom: 30px;
padding: 20px 24px;
background: #eef3f8;
border-left: 5px solid #0b82c5;
border-radius: 12px;
width: 100%;
}
.etl-legend__title {
font-size: 18px;
font-weight: 500;
color: #17253D;
width: 100%;
margin-bottom: 6px;
text-transform: uppercase;
letter-spacing: 0.03em;
}
.etl-legend__item {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 400;
color: #17253D;
}
.etl-legend__badge {
display: inline-flex;
align-items: center;
justify-content: center;
width: 34px;
height: 34px;
border-radius: 50%;
font-size: 18px;
font-weight: 600;
flex-shrink: 0;
}
.etl-legend__badge--star {
background: #fff4df;
color: var(--color-star);
}
.etl-legend__badge--check {
background: #e7f7ee;
color: var(--color-check);
}
.etl-legend__badge--half {
background: #fff8e8;
color: var(--color-partial);
}
.etl-legend__badge--cross {
background: #fdeeee;
color: var(--color-cross);
}
.etl-scroll-hint {
display: none;
font-size: 14px;
font-weight: 400;
color: #17253D;
margin-bottom: 8px;
text-align: right;
font-style: italic;
}
/* ===== Table Wrapper ===== */
.etl-table-wrapper {
width: 100%;
margin-bottom: 40px;
border-radius: var(--border-radius);
box-shadow: 0 2px 12px rgba(0,0,0,0.08);
overflow-x: visible;
}
/* ===== Main Table ===== */
.etl-table {
width: 100%;
min-width: 0;
table-layout: fixed;
border-collapse: collapse;
font-family: var(--font-family);
font-size: 18px;
font-weight: 400;
color: #17253D;
background: var(--color-bg-white);
}
/* Desktop column width balance */
.etl-table colgroup col:nth-child(1) { width: 24%; }
.etl-table colgroup col:nth-child(2) { width: 10%; }
.etl-table colgroup col:nth-child(3) { width: 10%; }
.etl-table colgroup col:nth-child(4) { width: 10%; }
.etl-table colgroup col:nth-child(5) { width: 46%; }
.etl-table thead tr {
background: var(--color-bg-header);
}
.etl-table thead th {
padding: 14px 10px;
color: #ffffff;
font-weight: 500;
font-size: 18px;
text-align: left;
border: var(--table-border);
border-color: rgba(255,255,255,0.12);
line-height: 1.35;
word-break: normal;
overflow-wrap: normal;
}
.etl-table thead th.tool-col {
text-align: center;
white-space: normal;
word-break: normal;
overflow-wrap: normal;
}
.etl-head-nowrap {
display: inline-block;
white-space: normal;
word-break: normal;
overflow-wrap: normal;
}
.etl-table tr.etl-cat-row td {
background: var(--color-bg-subheader);
color: #ffffff;
font-weight: 500;
font-size: 18px;
text-transform: uppercase;
letter-spacing: 0.03em;
padding: 12px 10px;
border: var(--table-border);
border-color: rgba(255,255,255,0.18);
}
.etl-table tbody tr.etl-data-row:nth-child(even) {
background: #ffffff;
}
.etl-table tbody tr.etl-data-row:hover {
background: var(--color-bg-alt);
}
.etl-table tbody tr.etl-data-row td {
padding: 13px 10px;
border: var(--table-border);
vertical-align: middle;
line-height: 1.45;
word-break: normal;
overflow-wrap: break-word;
font-size: 18px;
font-weight: 400;
color: #17253D;
}
.etl-table tbody tr.etl-data-row td:first-child {
font-size: 18px;
font-weight: 400;
color: #17253D;
}
.etl-table tbody tr.etl-data-row td:nth-child(2),
.etl-table tbody tr.etl-data-row td:nth-child(3),
.etl-table tbody tr.etl-data-row td:nth-child(4) {
text-align: center;
vertical-align: middle;
white-space: normal;
}
.etl-table tbody tr.etl-data-row td:last-child {
font-size: 18px;
font-weight: 400;
color: #17253D;
line-height: 1.45;
word-break: normal;
overflow-wrap: break-word;
}
.sym-star,
.sym-check,
.sym-partial,
.sym-cross {
display: inline-block;
font-size: 18px;
font-weight: 600;
line-height: 1;
}
.sym-star { color: var(--color-star); }
.sym-check { color: var(--color-check); }
.sym-partial { color: var(--color-partial); }
.sym-cross { color: var(--color-cross); }
.sym-text {
font-size: 18px;
font-weight: 400;
color: #17253D;
display: inline-block;
line-height: 1.3;
white-space: normal;
}
/* ===== Laptop / Desktop up to 1440px ===== */
@media (min-width: 1025px) and (max-width: 1440px) {
.etl-table {
width: 100%;
min-width: 0;
table-layout: fixed;
font-size: 18px;
}
.etl-table colgroup col:nth-child(1) { width: 23%; }
.etl-table colgroup col:nth-child(2) { width: 10%; }
.etl-table colgroup col:nth-child(3) { width: 10%; }
.etl-table colgroup col:nth-child(4) { width: 9%; }
.etl-table colgroup col:nth-child(5) { width: 48%; }
.etl-table thead th,
.etl-table tbody tr.etl-data-row td,
.etl-table tbody tr.etl-data-row td:first-child,
.etl-table tbody tr.etl-data-row td:last-child,
.etl-table tr.etl-cat-row td,
.sym-text {
font-size: 15px;
}
.etl-table thead th {
padding: 13px 8px;
}
.etl-table tbody tr.etl-data-row td {
padding: 12px 8px;
line-height: 1.42;
}
}
/* ===== Tablet ===== */
@media (max-width: 1024px) {
.etl-section {
padding: 0 12px;
}
.etl-scroll-hint {
display: block;
}
.etl-table-wrapper {
overflow-x: auto;
-webkit-overflow-scrolling: touch;
}
.etl-legend {
gap: 12px;
padding: 16px 18px;
margin-bottom: 20px;
}
.etl-table {
min-width: 1160px;
}
.etl-table colgroup col:nth-child(1) { width: 22%; }
.etl-table colgroup col:nth-child(2) { width: 14%; }
.etl-table colgroup col:nth-child(3) { width: 14%; }
.etl-table colgroup col:nth-child(4) { width: 12%; }
.etl-table colgroup col:nth-child(5) { width: 38%; }
.etl-table,
.etl-table thead th,
.etl-table tbody tr.etl-data-row td,
.etl-table tbody tr.etl-data-row td:first-child,
.etl-table tbody tr.etl-data-row td:last-child,
.etl-table tr.etl-cat-row td,
.etl-legend__title,
.etl-legend__item,
.sym-text {
font-size: 16px;
}
.etl-table thead th.tool-col,
.etl-head-nowrap {
white-space: nowrap;
}
.sym-star,
.sym-check,
.sym-partial,
.sym-cross {
font-size: 16px;
}
}
/* ===== Mobile ===== */
@media (max-width: 767px) {
.etl-section {
padding: 0 10px;
}
.etl-legend {
flex-direction: column;
gap: 8px;
padding: 14px 14px;
border-radius: 10px;
}
.etl-scroll-hint {
display: block;
}
.etl-table-wrapper {
overflow-x: auto;
-webkit-overflow-scrolling: touch;
}
.etl-table {
min-width: 1080px;
}
.etl-table,
.etl-table thead th,
.etl-table tbody tr.etl-data-row td,
.etl-table tbody tr.etl-data-row td:first-child,
.etl-table tbody tr.etl-data-row td:last-child,
.etl-table tr.etl-cat-row td,
.etl-legend__title,
.etl-legend__item,
.sym-text {
font-size: 14px;
}
.etl-table thead th {
padding: 10px 8px;
}
.etl-table tbody tr.etl-data-row td {
padding: 10px 8px;
}
.sym-star,
.sym-check,
.sym-partial,
.sym-cross {
font-size: 14px;
}
.etl-legend__badge {
width: 30px;
height: 30px;
font-size: 16px;
}
}
</style>
<div class="etl-section">
<div class="etl-legend">
<div class="etl-legend__title">Legend</div>
<div class="etl-legend__item">
<span class="etl-legend__badge etl-legend__badge--star">★</span>
<span>Unique / standout feature</span>
</div>
<div class="etl-legend__item">
<span class="etl-legend__badge etl-legend__badge--check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span>
<span>Strong / full support</span>
</div>
<div class="etl-legend__item">
<span class="etl-legend__badge etl-legend__badge--half">◐</span>
<span>Partial / limited support</span>
</div>
<div class="etl-legend__item">
<span class="etl-legend__badge etl-legend__badge--cross">✘</span>
<span>Not supported / not available</span>
</div>
</div>
<p class="etl-scroll-hint">← Scroll to see full table →</p>
<div class="etl-table-wrapper">
<table class="etl-table">
<colgroup>
<col>
<col>
<col>
<col>
<col>
</colgroup>
<thead>
<tr>
<th>Feature / Capability</th>
<th class="tool-col"><span class="etl-head-nowrap">Datagaps<br>ETL Validator</span></th>
<th class="tool-col"><span class="etl-head-nowrap">QuerySurge</span></th>
<th class="tool-col"><span class="etl-head-nowrap">dbt Tests</span></th>
<th>Verdict</th>
</tr>
</thead>
<tbody>
<tr class="etl-cat-row"><td colspan="5">1. Core ETL Testing</td></tr>
<tr class="etl-data-row">
<td>ETL Test Authoring & Execution</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge are purpose-built for end-to-end ETL test authoring and execution. dbt Tests define quality checks on dbt models only.</td>
</tr>
<tr class="etl-data-row">
<td>ELT / In-Database Testing</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td>ETL Validator and dbt Tests push validation to the warehouse natively. ETL Validator leads on orchestration across multiple platforms. QuerySurge is partial.</td>
</tr>
<tr class="etl-data-row">
<td>Flat File / CSV Testing</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator and QuerySurge handle flat file and CSV validation natively. dbt Tests are database-only.</td>
</tr>
<tr class="etl-data-row">
<td>Multiple Source / Target Support</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator supports multiple heterogeneous sources and targets in a single test run. QuerySurge supports only a single source-target pair. dbt Tests operate within a single warehouse.</td>
</tr>
<tr class="etl-data-row">
<td>Transformation Validation</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td>ETL Validator adds GenAI-assisted rule authoring across any ecosystem. dbt Tests are strong for validating dbt model outputs. QuerySurge uses SQL-based validation.</td>
</tr>
<tr class="etl-data-row">
<td>Source-to-Target Reconciliation</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator uniquely supports Data Profile reconciliation. QuerySurge covers row counts and aggregations. dbt has no cross-system reconciliation.</td>
</tr>
<tr class="etl-data-row">
<td>Source-to-Report Testing</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator validates the full chain from raw source through to the BI report layer. QuerySurge has limited support. dbt Tests do not reach the reporting layer.</td>
</tr>
<tr class="etl-data-row">
<td>Non-dbt Pipeline Testing</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator and QuerySurge test any pipeline regardless of transformation tool. dbt Tests are locked to dbt models.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">2. Automation & CI/CD</td></tr>
<tr class="etl-data-row">
<td>Automated Regression Testing</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator adds GenAI-assisted test maintenance. QuerySurge offers structured ETL regression automation. dbt Tests re-run on every invocation but have no dedicated regression management.</td>
</tr>
<tr class="etl-data-row">
<td>CI/CD Pipeline Integration</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-star">★</span></td>
<td>dbt Tests have first-class CI/CD integration. ETL Validator and QuerySurge both support CI/CD with broad pipeline trigger options.</td>
</tr>
<tr class="etl-data-row">
<td>Scheduled / Triggered Test Runs</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge support native scheduling and REST API triggers. dbt Tests depend on dbt Cloud or an external orchestrator such as Airflow.</td>
</tr>
<tr class="etl-data-row">
<td>Test Case Reusability</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td>All three support reusable test definitions. ETL Validator and QuerySurge offer reusable templates via their UIs and test libraries.</td>
</tr>
<tr class="etl-data-row">
<td>Test Maintenance Overhead</td>
<td><span class="sym-text">Low</span></td>
<td><span class="sym-text">Medium</span></td>
<td><span class="sym-text">Medium-High</span></td>
<td>ETL Validator's GenAI-assisted maintenance significantly reduces upkeep as pipelines change. dbt Tests require engineers to update definitions manually for every model or schema change.</td>
</tr>
<tr class="etl-data-row">
<td>Cross-Pipeline Orchestration</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator and QuerySurge orchestrate tests across multiple pipelines in a single run. dbt Tests are scoped to the dbt DAG.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">3. Usability & Test Authoring</td></tr>
<tr class="etl-data-row">
<td>No-Code / Visual Test Builder</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator is the only tool with a drag-and-drop no-code interface for ETL testing. QuerySurge is partial. dbt Tests are written entirely in YAML and SQL.</td>
</tr>
<tr class="etl-data-row">
<td>Ease of Setup</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge deploy in days. dbt Tests require an existing dbt project before writing a single test.</td>
</tr>
<tr class="etl-data-row">
<td>Business User Accessibility</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator is designed for QA analysts and business users without coding skills. QuerySurge requires SQL knowledge. dbt Tests require proficiency in dbt, YAML, SQL, and version control.</td>
</tr>
<tr class="etl-data-row">
<td>GenAI / AI-Assisted Test Creation</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator generates tests automatically from ETL mapping documents using agentic AI, cutting initial test creation time by over 60%. QuerySurge offers limited GenAI support.</td>
</tr>
<tr class="etl-data-row">
<td>Test Documentation & Visibility</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator provides customisable stakeholder dashboards. QuerySurge offers detailed reporting. dbt generates docs automatically but test visibility for non-engineers is limited.</td>
</tr>
<tr class="etl-data-row">
<td>Learning Curve</td>
<td><span class="sym-text">Low</span></td>
<td><span class="sym-text">Low-Medium</span></td>
<td><span class="sym-text">High</span></td>
<td>ETL Validator is the fastest to productive use for any team profile. dbt Tests require mastery of the full dbt framework.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">4. Data Quality & Observability</td></tr>
<tr class="etl-data-row">
<td>Data Quality Monitoring</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator provides continuous DQ monitoring with scoring and alerting. dbt Tests and QuerySurge run at job execution time only.</td>
</tr>
<tr class="etl-data-row">
<td>Anomaly Detection</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator automatically detects data anomalies across pipelines using AI. Neither QuerySurge nor dbt Tests offer automated anomaly detection.</td>
</tr>
<tr class="etl-data-row">
<td>Data Profiling</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator provides rich data profiling alongside test execution. QuerySurge offers basic profiling. dbt Tests require separate tools such as dbt-profiler or Elementary.</td>
</tr>
<tr class="etl-data-row">
<td>Data Lineage</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-star">★</span></td>
<td>dbt auto-generates column-level lineage across the entire DAG. ETL Validator provides pipeline-level lineage tied to DQ scoring. QuerySurge has no lineage support.</td>
</tr>
<tr class="etl-data-row">
<td>DQ Scoring & Health Dashboards</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator uniquely provides quantified DQ scores and health dashboards across pipelines. Neither QuerySurge nor dbt offer this natively.</td>
</tr>
<tr class="etl-data-row">
<td>Alerting & Notifications</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge support native alerting on test failures. dbt alerting depends on the orchestration layer.</td>
</tr>
<tr class="etl-data-row">
<td>BI Regression Testing</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator's visual BI report regression testing across Power BI, Tableau, QuickSight, and Oracle Analytics has no equivalent in QuerySurge or dbt.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">5. Data Contracts & Governance</td></tr>
<tr class="etl-data-row">
<td>Data Contracts</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator supports formal data contracts for validating data and schema obligations across pipeline boundaries. dbt has partial support via dbt contracts. QuerySurge has none.</td>
</tr>
<tr class="etl-data-row">
<td>Schema Validation & Drift Detection</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td>ETL Validator and dbt Tests both detect schema drift. QuerySurge offers partial schema validation.</td>
</tr>
<tr class="etl-data-row">
<td>Data Observability Integration</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator provides built-in observability across the full pipeline. dbt integrates with third-party tools. QuerySurge is less observability-focused.</td>
</tr>
<tr class="etl-data-row">
<td>Audit Trails & Compliance Reporting</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator and QuerySurge provide compliance-grade audit trails out of the box. dbt requires significant custom engineering to produce audit-ready reports.</td>
</tr>
<tr class="etl-data-row">
<td>Role-Based Access Control</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge support enterprise RBAC natively. dbt Cloud offers team-level permissions; dbt Core has no access control layer.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">6. Testing Scope & Coverage</td></tr>
<tr class="etl-data-row">
<td>Mixed-Source Pipelines</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator's Apache Spark engine supports heterogeneous sources including databases, files, and APIs. dbt is warehouse-only.</td>
</tr>
<tr class="etl-data-row">
<td>Legacy System Testing</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator and QuerySurge test pipelines built in any ETL tool including legacy platforms. dbt Tests are not suitable for non-dbt pipelines.</td>
</tr>
<tr class="etl-data-row">
<td>Streaming / Real-Time Data Validation</td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator and QuerySurge have partial streaming support. dbt is mainly a batch transformation tool.</td>
</tr>
<tr class="etl-data-row">
<td>Extensibility</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator provides the capability to add custom plugins using Python, making it highly extensible. QuerySurge and dbt have a fixed set of capabilities.</td>
</tr>
<tr class="etl-data-row">
<td>Test Data Generation</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-cross">✘</span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator uniquely generates synthetic test data for automating pipeline testing, reducing reliance on production data copies.</td>
</tr>
<tr class="etl-data-row">
<td>End-to-End Pipeline Coverage</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator covers ingestion, transformation, loading, and BI reporting. dbt Tests cover only the transformation layer within dbt models.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">7. Enterprise Readiness</td></tr>
<tr class="etl-data-row">
<td>Enterprise Support & SLAs</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge offer dedicated commercial support with SLAs. dbt Core is open-source with community support only.</td>
</tr>
<tr class="etl-data-row">
<td>On-Premise Deployment</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator and QuerySurge support on-premise deployment. dbt Cloud is SaaS-based.</td>
</tr>
<tr class="etl-data-row">
<td>Multi-Project / Multi-Team Support</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator supports multiple projects in a single deployment with container isolation. QuerySurge supports multi-team setups.</td>
</tr>
<tr class="etl-data-row">
<td>Custom Dashboards for Stakeholders</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-cross">✘</span></td>
<td>ETL Validator uniquely provides customisable stakeholder-facing dashboards for sharing test results and data quality scores.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">8. Scalability & Performance</td></tr>
<tr class="etl-data-row">
<td>Handling Large Data Volumes</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td>ETL Validator's Spark-based execution engine is built for billions of records. QuerySurge is comparatively limited for enterprise-scale data volumes.</td>
</tr>
<tr class="etl-data-row">
<td>Auto-Scaling</td>
<td><span class="sym-star">★</span></td>
<td><span class="sym-partial">◐</span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator has native on-demand auto-scaling. dbt and QuerySurge rely on underlying infrastructure.</td>
</tr>
<tr class="etl-data-row">
<td>Parallel Test Execution</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-partial">◐</span></td>
<td>ETL Validator's Spark engine enables high-parallelism across hundreds of tests simultaneously. dbt test parallelism is warehouse-dependent.</td>
</tr>
<tr class="etl-data-row">
<td>Cloud-Native Deployment</td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td><span class="sym-check"><img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2714.png" alt="✔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></span></td>
<td>All three are cloud-native. ETL Validator supports AKS, EKS, GKE, and Databricks. dbt Cloud is fully managed.</td>
</tr>
<tr class="etl-cat-row"><td colspan="5">9. Pricing & Accessibility</td></tr>
<tr class="etl-data-row">
<td>Licensing Model</td>
<td><span class="sym-text">Commercial</span></td>
<td><span class="sym-text">Commercial</span></td>
<td><span class="sym-text">Open-Source / dbt Cloud</span></td>
<td>dbt Core is free and open-source; dbt Cloud adds a managed commercial tier. The true cost of dbt Tests includes engineering time to build, maintain, and extend.</td>
</tr>
<tr class="etl-data-row">
<td>Relative Cost</td>
<td><span class="sym-text">Best value</span></td>
<td><span class="sym-text">Mid-range</span></td>
<td><span class="sym-text">Free + engineering cost</span></td>
<td>dbt Tests appear free, but the hidden cost is engineering hours to configure and maintain them. ETL Validator delivers broad feature coverage across total cost of ownership.</td>
</tr>
<tr class="etl-data-row">
<td>ETL Vendor Lock-in Risk</td>
<td><span class="sym-text">Low</span></td>
<td><span class="sym-text">Low</span></td>
<td><span class="sym-text">Medium</span></td>
<td>dbt Tests are tightly coupled to the dbt ecosystem. ETL Validator and QuerySurge carry low lock-in risk.</td>
</tr>
<tr class="etl-data-row">
<td>Ideal Team Profile</td>
<td><span class="sym-text">Data Engineering & QA teams of all sizes</span></td>
<td><span class="sym-text">QA Teams</span></td>
<td><span class="sym-text">dbt-native analytics engineers</span></td>
<td>dbt Tests only make sense for teams already running dbt. ETL Validator serves QA, engineering, and business users.</td>
</tr>
</tbody>
</table>
</div>
</div> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-f061c92 e-flex e-con-boxed e-con e-parent" data-id="f061c92" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-83d85be elementor-widget elementor-widget-heading" data-id="83d85be" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Which ETL Testing Tool Should You Choose?</h3> </div>
</div>
<div class="elementor-element elementor-element-7caf13f elementor-widget elementor-widget-text-editor" data-id="7caf13f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="font-claude-response-body">Choosing the right <span style="color: #1967d2;"><a style="color: #1967d2;" href="https://www.datagaps.com/data-validation-etl-testing-tools/" target="_blank" rel="noopener"><span style="text-decoration: underline;">ETL testing tool</span></a></span> depends on how comprehensive your testing needs are across data pipelines. While multiple tools offer specific capabilities, they differ significantly in scope, flexibility, and coverage.</p> </div>
</div>
<div class="elementor-element elementor-element-721f3b4 elementor-position-inline-start elementor-mobile-position-inline-start elementor-view-default elementor-widget elementor-widget-icon-box" data-id="721f3b4" data-element_type="widget" data-e-type="widget" data-widget_type="icon-box.default">
<div class="elementor-widget-container">
<div class="elementor-icon-box-wrapper">
<div class="elementor-icon-box-icon">
<span class="elementor-icon">
<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32"><g id="Group_20826" data-name="Group 20826" transform="translate(-4197 14921)"><g id="Group_601" data-name="Group 601" transform="translate(4197 -14921)"><circle id="Ellipse_30" data-name="Ellipse 30" cx="16" cy="16" r="16" fill="#1eb473"></circle><path id="Path_426" data-name="Path 426" d="M4732.163-15573.172l4.563,4.191,8.547-9.346" transform="translate(-4722.81 15589.505)" fill="none" stroke="#fff" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"></path></g></g></svg> </span>
</div>
<div class="elementor-icon-box-content">
<h4 class="elementor-icon-box-title">
<span >
Datagaps ETL Validator </span>
</h4>
<p class="elementor-icon-box-description">
Datagaps ETL Validator provides a more complete approach by supporting end-to-end ETL testing across heterogeneous data sources, including databases, files, APIs and BI layers. It also offers automation, AI-driven test generation, and scalability required for modern data environments. </p>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-7fad8bc elementor-position-inline-start elementor-mobile-position-inline-start elementor-view-default elementor-widget elementor-widget-icon-box" data-id="7fad8bc" data-element_type="widget" data-e-type="widget" data-widget_type="icon-box.default">
<div class="elementor-widget-container">
<div class="elementor-icon-box-wrapper">
<div class="elementor-icon-box-icon">
<span class="elementor-icon">
<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32"><g id="Group_20826" data-name="Group 20826" transform="translate(-4197 14921)"><g id="Group_601" data-name="Group 601" transform="translate(4197 -14921)"><circle id="Ellipse_30" data-name="Ellipse 30" cx="16" cy="16" r="16" fill="#1eb473"></circle><path id="Path_426" data-name="Path 426" d="M4732.163-15573.172l4.563,4.191,8.547-9.346" transform="translate(-4722.81 15589.505)" fill="none" stroke="#fff" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"></path></g></g></svg> </span>
</div>
<div class="elementor-icon-box-content">
<h4 class="elementor-icon-box-title">
<span >
QuerySurge </span>
</h4>
<p class="elementor-icon-box-description">
QuerySurge is effective for SQL-based validation but is largely limited to query-pair comparisons and does not support broader multi-system or end-to-end pipeline testing scenarios. </p>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-b287efa elementor-position-inline-start elementor-mobile-position-inline-start elementor-view-default elementor-widget elementor-widget-icon-box" data-id="b287efa" data-element_type="widget" data-e-type="widget" data-widget_type="icon-box.default">
<div class="elementor-widget-container">
<div class="elementor-icon-box-wrapper">
<div class="elementor-icon-box-icon">
<span class="elementor-icon">
<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32"><g id="Group_20826" data-name="Group 20826" transform="translate(-4197 14921)"><g id="Group_601" data-name="Group 601" transform="translate(4197 -14921)"><circle id="Ellipse_30" data-name="Ellipse 30" cx="16" cy="16" r="16" fill="#1eb473"></circle><path id="Path_426" data-name="Path 426" d="M4732.163-15573.172l4.563,4.191,8.547-9.346" transform="translate(-4722.81 15589.505)" fill="none" stroke="#fff" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"></path></g></g></svg> </span>
</div>
<div class="elementor-icon-box-content">
<h4 class="elementor-icon-box-title">
<span >
dbt tests </span>
</h4>
<p class="elementor-icon-box-description">
dbt Tests are limited to rule-based data checks within a single data warehouse. They are not built for complete ETL testing and do not address pipeline validation across systems. </p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-81d653a e-flex e-con-boxed e-con e-parent" data-id="81d653a" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-e44de72 elementor-widget elementor-widget-heading" data-id="e44de72" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Our Recommendation for ETL Testing Tool</h3> </div>
</div>
<div class="elementor-element elementor-element-c7fb04b elementor-widget elementor-widget-text-editor" data-id="c7fb04b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="font-weight: 600;">For teams that need comprehensive coverage across the full pipeline, <span style="text-decoration: underline; color: #1967d2;"><a style="color: #1967d2; font-weight: 600;" href="https://www.datagaps.com/data-validation-etl-testing-tools/" target="_blank" rel="noopener">Datagaps ETL Validator </a></span>is the clear choice. </span>Where QuerySurge stops at query-pair validation and does not scale effectively for large data volumes, and dbt Tests stay within the warehouse running rule-based checks, Datagaps ETL Validator goes further: across sources, through transformations, and all the way to the BI reporting layer. Built on a Spark-based engine, Datagaps ETL Validator is designed to scale for enterprise data volumes without compromising on performance. It is purpose-built for ETL testing and Datagaps is recognized as a data pipelines test automation specialist in Gartner’s Market Guide for DataOps Tools. If reliable, end-to-end data validation matters to your team, Datagaps ETL Validator is the tool built for that job.</p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-cb60e84 e-flex e-con-boxed e-con e-parent" data-id="cb60e84" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-d0aed99 elementor-widget elementor-widget-text-editor" data-id="d0aed99" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>For teams looking beyond framework-specific validation toward complete pipeline testing and ETL automation, <span style="text-decoration: underline;"><span style="color: #1967d2;"><strong><a style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/data-validation-etl-testing-tools/" target="_blank" rel="noopener">Datagaps ETL Validator</a></strong></span></span> offers a more comprehensive approach.</p> </div>
</div>
<div class="elementor-element elementor-element-adf490a elementor-widget elementor-widget-text-editor" data-id="adf490a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="text-decoration: underline;">Disclaimer</span>: The above-mentioned list is purely an outcome of the conversations and feedback received from various industry users in the ETL/Data Warehouse testing space. Any concerns or views can be shared at <span style="text-decoration: underline; color: #1967d2;"><a style="color: #1967d2; text-decoration: underline;" href="mailto:contact@datagaps.com">contact@datagaps.com</a></span></p> </div>
</div>
<div class="elementor-element elementor-element-763e58e elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="763e58e" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-a50e9c0 e-con-full e-flex e-con e-child" data-id="a50e9c0" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-f962a40 e-con-full e-flex e-con e-child" data-id="f962a40" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="elementor-element elementor-element-590215f e-con-full e-flex e-con e-child" data-id="590215f" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-094370d elementor-widget elementor-widget-heading" data-id="094370d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Watch ETL Validator in Action with Demo</h2> </div>
</div>
<div class="elementor-element elementor-element-ac1d03e elementor-widget elementor-widget-text-editor" data-id="ac1d03e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Check out how ETL Validator simplifies ETL Testing, data validation through automation across pipelines from this playlist </div>
</div>
</div>
<div class="elementor-element elementor-element-f0c0932 e-con-full e-flex e-con e-child" data-id="f0c0932" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-3adea3d premium-lq__none elementor-widget elementor-widget-premium-addon-button" data-id="3adea3d" data-element_type="widget" data-e-type="widget" data-widget_type="premium-addon-button.default">
<div class="elementor-widget-container">
<a class="premium-button premium-button-none premium-btn-md premium-button-none" href="https://www.youtube.com/playlist?list=PLq-Q4hhL4wuA7vizbNdbV_dVI-3vyacaI">
<div class="premium-button-text-icon-wrapper">
<span >
Demo Playlist </span>
</div>
</a>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-2d0076b elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="2d0076b" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-9307294 e-con-full e-flex e-con e-child" data-id="9307294" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="elementor-element elementor-element-de8f35d e-con-full e-flex e-con e-child" data-id="de8f35d" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-561423e elementor-widget elementor-widget-text-editor" data-id="561423e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Start your 14-day free trial in our sandbox. Explore and optimize your ETL processes. Start your trial today!</p> </div>
</div>
<div class="elementor-element elementor-element-e0b53cc elementor-widget elementor-widget-heading" data-id="e0b53cc" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Get Started with ETL Validator – An ETL & Data Testing tool</h2> </div>
</div>
</div>
<div class="elementor-element elementor-element-3eb19f1 e-con-full e-flex e-con e-child" data-id="3eb19f1" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-3371474 premium-lq__none elementor-widget elementor-widget-premium-addon-button" data-id="3371474" data-element_type="widget" data-e-type="widget" data-widget_type="premium-addon-button.default">
<div class="elementor-widget-container">
<a class="premium-button premium-button-none premium-btn-md premium-button-none" href="https://www.datagaps.com/request-a-demo/">
<div class="premium-button-text-icon-wrapper">
<span >
Request a Demo </span>
</div>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/top-3-etl-testing-tools/">Top 3 ETL Testing Tools: How to Choose the Best Tool</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></content:encoded>
</item>
<item>
<title>ETL Testing In Snowflake Using DataOps Suite</title>
<link>https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/</link>
<comments>https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/#respond</comments>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 14 Feb 2023 13:18:56 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=11775</guid>
<description><![CDATA[<p>ETL stands for Extract, Transform, and Load. It is the process by which data is extracted from one or more sources, transformed into compatible formats, and then loaded into a target Database or Data Warehouse.</p>
<p>The post <a href="https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/">ETL Testing In Snowflake Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="11775" class="elementor elementor-11775" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Introduction and Overview of ETL Testing Snowflake</h2> </div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.ibm.com/in-en/topics/etl">ETL</a> stands for Extract, Transform, and Load. It is the process by which data is extracted from one or more sources, transformed into compatible formats, and then loaded into a target Database or Data Warehouse. The sources may include Flat Files, Third-Party Applications, Databases, etc. <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> is necessary to ensure that data moving from external sources to the data warehouse is accurate at each point between the source and destination.</p> </div>
</div>
<div class="elementor-element elementor-element-73a039c elementor-widget elementor-widget-heading" data-id="73a039c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Purpose of ETL</h3> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL allows businesses to consolidate data from multiple databases and other sources into a single repository with the data that has been modified and used during the analysis of data. This unified data repository allows for simplified access to analysis and additional processing of the data. There are many advantages of using ETL tools for the migration of data. It reduces delivery time, reduces unnecessary expenses, makes the process easy to use, and also will be simple for data migrations. Data Integration, Data Warehousing, and Data Migration are the three common uses of ETL.</p> </div>
</div>
<div class="elementor-element elementor-element-dd5971f elementor-widget elementor-widget-heading" data-id="dd5971f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">ETL Testing Process in Snowflake</h3> </div>
</div>
<div class="elementor-element elementor-element-4bfe18a elementor-widget elementor-widget-text-editor" data-id="4bfe18a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The data will be migrated from one data warehouse to another cloud-based <a href="https://aws.amazon.com/data-warehouse/">data warehouse</a> using various steps present in ETL Testing. The multiple steps involved in this process are the extraction of data, the transformation of the data, and finally the loading of data to the different data sources. This process is essential for proper testing such the quality of data can be checked efficiently. The DataOps Suite tool can be used efficiently for ETL Testing. <a href="https://www.datagaps.com/request-demo/">Request Demo</a></p> </div>
</div>
<div class="elementor-element elementor-element-39a261d elementor-widget elementor-widget-text-editor" data-id="39a261d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<h2><strong>The various steps involved in ETL Testing are as follows:</strong></h2> </div>
</div>
<div class="elementor-element elementor-element-27ffd33 elementor-widget elementor-widget-heading" data-id="27ffd33" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Step 1: Extraction Of Data</h3> </div>
</div>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data Extraction is the first step that will be performed in the ETL Testing. In this procedure, the data will usually be extracted from the same data source, or it can be extracted from different source locations also. Here, for example, the data is extracted from the same source i.e. Snowflake, and Customer data is extracted. After extracting the data from the source location, then further the data can be transformed according to the client’s requirements.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-2ea2482 elementor-widget elementor-widget-image" data-id="2ea2482" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img fetchpriority="high" decoding="async" width="1320" height="697" src="https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table.png" class="attachment-full size-full wp-image-11776" alt="Data-Extraction-From-Customers-Table" srcset="https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table.png 1320w, https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table-1024x541.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table-768x406.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Extraction From Customers Table</figcaption>
</figure>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-deb51bb elementor-widget elementor-widget-heading" data-id="deb51bb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Step 2: Transformation Of Data</h3> </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>After the data is extracted from the same or different data source to the same or the other source, a few changes or transformations in the customers’ data are done. Generally, data transformations include changes in data types or other changes according to the client’s requirements.</p><p>The below screenshot depicts the Customer data that is being transformed.</p> </div>
</div>
<div class="elementor-element elementor-element-e4db7c6 elementor-widget elementor-widget-image" data-id="e4db7c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img decoding="async" width="1321" height="695" src="https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component.png" class="attachment-full size-full wp-image-11777" alt="Data-Transformation-Using-SQL-Component" srcset="https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component.png 1321w, https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component-768x404.png 768w" sizes="(max-width: 1321px) 100vw, 1321px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Transformation Using SQL Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-324bb87 elementor-widget elementor-widget-text-editor" data-id="324bb87" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once the data is transformed, <strong>data comparison </strong>can be performed to view the changes after transformation.</p> </div>
</div>
<div class="elementor-element elementor-element-62f0ab3 elementor-widget elementor-widget-image" data-id="62f0ab3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img decoding="async" width="1320" height="700" src="https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component.png" class="attachment-full size-full wp-image-11778" alt="Comparison-Of-Data-Using-Data-Compare-Component" srcset="https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component.png 1320w, https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component-300x159.png 300w, https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component-1024x543.png 1024w, https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component-768x407.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Comparison Of Data Using Data Compare Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="fontSizeMediumPlus">Further, the quality of data can be checked by using the <strong>Data Rules Component. </strong></span><span class="fontSizeMediumPlus">Data quality checks are done to find out the issues in the quality of data. The <strong>Data</strong><strong> Profile Component</strong> can also be used to find out the data quality issues.</span></p> </div>
</div>
<div class="elementor-element elementor-element-2955a70 elementor-widget elementor-widget-text-editor" data-id="2955a70" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the below screenshot, the quality of data is checked by verifying the email address as well as the name string check by using different data rules in the data rules component.</p> </div>
</div>
<div class="elementor-element elementor-element-066456a elementor-widget elementor-widget-image" data-id="066456a" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1323" height="696" src="https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component.png" class="attachment-full size-full wp-image-11779" alt="Data-Quality-Check-Using-Data-Rules-Component" srcset="https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component.png 1323w, https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component-768x404.png 768w" sizes="(max-width: 1323px) 100vw, 1323px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Quality Check Using Data Rules Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-fc1846b elementor-widget elementor-widget-text-editor" data-id="fc1846b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data profiling is also done to check the quality of data.</p> </div>
</div>
<div class="elementor-element elementor-element-0163057 elementor-widget elementor-widget-image" data-id="0163057" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1320" height="695" src="https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component.png" class="attachment-full size-full wp-image-11780" alt="Profiling-Data-Using-Data-Profile-Component" srcset="https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component.png 1320w, https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component-768x404.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Profiling Data Using Data Profile Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-9c4aaae elementor-widget elementor-widget-heading" data-id="9c4aaae" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Step 3: Loading Of The Data</h3> </div>
</div>
<div class="elementor-element elementor-element-a757e12 elementor-widget elementor-widget-text-editor" data-id="a757e12" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once the transformation of data is performed, further the data will be loaded from one source to a particular file location. Here the data is loaded by using the <strong>DB Sink component.</strong> This is the general testing process followed in the DataOps Suite tool. <a href="https://www.datagaps.com/request-demo/">Request Demo</a></p> </div>
</div>
<div class="elementor-element elementor-element-d042b15 elementor-widget elementor-widget-text-editor" data-id="d042b15" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The below screenshot depicts the data loaded to the desired data source after the data transformations are done.</p> </div>
</div>
<div class="elementor-element elementor-element-d232146 elementor-widget elementor-widget-image" data-id="d232146" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1320" height="695" src="https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component.png" class="attachment-full size-full wp-image-11781" alt="Data-Loading-Using-DB-Sink-Component" srcset="https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component.png 1320w, https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component-768x404.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Loading Using DB Sink Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-e5f9257 elementor-widget elementor-widget-text-editor" data-id="e5f9257" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once the ETL Testing process is completed, the reports generated need to be checked and evaluated as there will be some differences. In our DataOps Suite tool, BI Validator can be used to check and evaluate the reports.</p><p> </p><p><a href="https://www.datagaps.com/data-testing-concepts/data-warehouse-testing-checklist/">Read: Data Warehouse Testing Checklist</a></p> </div>
</div>
<div class="elementor-element elementor-element-e3a805c elementor-widget elementor-widget-heading" data-id="e3a805c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Conclusion</h3> </div>
</div>
<div class="elementor-element elementor-element-80ad1bd elementor-widget elementor-widget-text-editor" data-id="80ad1bd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> is an important process when data is transferred from one or multiple databases to another database, especially when a huge amount of data is used. It makes sure that the data loaded in the destination source is accurate enough. The step-by-step procedure of ETL Testing can be checked by using different components in our DataOps Suite tool. By using ETL Testing, the performance can be increased. Once the entire ETL Testing is completed in Snowflake, then finally the reports will be generated. The reports generated will be checked and validated finally by using the <a href="https://www.datagaps.com/bi-validator/">BI Validator</a>.</p> </div>
</div>
<div class="elementor-element elementor-element-287ddfd elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="287ddfd" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/">ETL Testing In Snowflake Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></content:encoded>
<wfw:commentRss>https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Data Validation for Snowflake</title>
<link>https://www.datagaps.com/blog/snowflake-data-migration-testing/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Fri, 18 Feb 2022 14:28:42 +0000</pubDate>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7365</guid>
<description><![CDATA[<p>Migration from a legacy data warehouse such as Netezza to a cloud-based Snowflake data warehouse requires multiple steps. Data Validation is the key to the success of data migration projects. Datagaps Data Flow can be used to validate data in each step of the data migration as</p>
<p>The post <a href="https://www.datagaps.com/blog/snowflake-data-migration-testing/">Data Validation for Snowflake</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7365" class="elementor elementor-7365" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-4431704 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="4431704" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-394eed7" data-id="394eed7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-b9c266f elementor-widget__width-inherit elementor-widget elementor-widget-image" data-id="b9c266f" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="254" src="https://www.datagaps.com/wp-content/uploads/snowflake-data-migration-1-1024x407.webp" class="attachment-large size-large wp-image-6390" alt="snowflake-data-migration-1" srcset="https://www.datagaps.com/wp-content/uploads/snowflake-data-migration-1-1024x407.webp 1024w, https://www.datagaps.com/wp-content/uploads/snowflake-data-migration-1-300x119.webp 300w, https://www.datagaps.com/wp-content/uploads/snowflake-data-migration-1-768x305.webp 768w, https://www.datagaps.com/wp-content/uploads/snowflake-data-migration-1-1536x611.webp 1536w, https://www.datagaps.com/wp-content/uploads/snowflake-data-migration-1.webp 1798w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-6d1fadc elementor-widget elementor-widget-heading" data-id="6d1fadc" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Migrating to Snowflake Cloud Data Warehouse</h5> </div>
</div>
<div class="elementor-element elementor-element-725e865 elementor-widget elementor-widget-text-editor" data-id="725e865" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Migration from a legacy data warehouse such as Netezza to a cloud-based <a href="https://www.snowflake.com/en/fundamentals/data-warehouse/">Snowflake data warehouse</a> requires multiple steps. Data Validation is the key to the success of data migration projects. Datagaps DataFlow can be used to validate data in each step of the data migration as well as the end-to-end data validation scenarios. If you are looking for Snowflake testing tools – <a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try Dataflow free for 14 days</a> </p> </div>
</div>
<div class="elementor-element elementor-element-9301940 elementor-widget elementor-widget-heading" data-id="9301940" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Step 1: Extract data from the Legacy data warehouse</h5> </div>
</div>
<div class="elementor-element elementor-element-caaebe3 elementor-widget elementor-widget-text-editor" data-id="caaebe3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data is typically extracted into CSV or Parquet format and moved to a landing zone in AWS S3. Depending on the data volumes, AWS offers multiple options for moving the files to S3. Once the data has been moved to AWS S3, data validations need to be performed to ensure that all the data was properly extracted and migrated to AWS S3. Since there are not many transformations in this step, these tests are typically one-to-one comparisons of the data in the tables in the legacy data warehouse and the files in the AWS S3 landing zone.</p> </div>
</div>
<div class="elementor-element elementor-element-c83d67a elementor-widget elementor-widget-text-editor" data-id="c83d67a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>– Compare table to file row counts<br />– Compare data encoding<br />– Compare data completeness<br />– Compare data values</p> </div>
</div>
<div class="elementor-element elementor-element-7433790 elementor-widget elementor-widget-text-editor" data-id="7433790" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A sample test case diagram is shown to the right. JDBC Component can be used to read data from the legacy data warehouse. File Component can be used to read data from AWS S3. Finally, the Data Compare component can be used to compare the two datasets. Sample output for a data comparison component is shown below.</p> </div>
</div>
<div class="elementor-element elementor-element-96b1a88 elementor-widget__width-inherit elementor-widget elementor-widget-image" data-id="96b1a88" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="584" height="219" src="https://www.datagaps.com/wp-content/uploads/oracle-to-S3.webp" class="attachment-large size-large wp-image-6400" alt="oracle-to-S3" srcset="https://www.datagaps.com/wp-content/uploads/oracle-to-S3.webp 584w, https://www.datagaps.com/wp-content/uploads/oracle-to-S3-300x113.webp 300w" sizes="(max-width: 584px) 100vw, 584px" /> </div>
</div>
<div class="elementor-element elementor-element-b40f4b3 elementor-widget elementor-widget-spacer" data-id="b40f4b3" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-5cf82e2 elementor-widget elementor-widget-text-editor" data-id="5cf82e2" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data comparison test case</p> </div>
</div>
<div class="elementor-element elementor-element-20a50bd elementor-widget__width-inherit elementor-widget elementor-widget-image" data-id="20a50bd" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="327" src="https://www.datagaps.com/wp-content/uploads/data-compare.webp" class="attachment-large size-large wp-image-6402" alt="data-compare" srcset="https://www.datagaps.com/wp-content/uploads/data-compare.webp 1000w, https://www.datagaps.com/wp-content/uploads/data-compare-300x153.webp 300w, https://www.datagaps.com/wp-content/uploads/data-compare-768x392.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-8e91434 elementor-widget elementor-widget-spacer" data-id="8e91434" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-a01b434 elementor-widget elementor-widget-text-editor" data-id="a01b434" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The output of data comparison</p> </div>
</div>
<div class="elementor-element elementor-element-bd58283 elementor-widget elementor-widget-heading" data-id="bd58283" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Step 2: Transform data</h5> </div>
</div>
<div class="elementor-element elementor-element-80a5735 elementor-widget elementor-widget-text-editor" data-id="80a5735" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Transformations such as data type conversions can be performed in this step. Data curation can be also done to improve the <a href="https://www.datagaps.com/dataops-suite/data-quality/">Data Quality</a> before the data is loaded into Snowflake. Before curating the data, it is important to profile the data and run data quality tests to identify data quality issues with the data. DataFlow can be used to perform these tasks.</p> </div>
</div>
<div class="elementor-element elementor-element-979de10 elementor-widget elementor-widget-text-editor" data-id="979de10" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>– Compare data between landing zone and staging (curated) zone in S3<br />– Use Data Profile and Data Rules components to identify data quality issues<br />– Curate data and sync to the staging zone</p> </div>
</div>
<div class="elementor-element elementor-element-89766c7 elementor-widget__width-inherit elementor-widget elementor-widget-image" data-id="89766c7" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="398" src="https://www.datagaps.com/wp-content/uploads/data-profile.webp" class="attachment-large size-large wp-image-6412" alt="data-profile" srcset="https://www.datagaps.com/wp-content/uploads/data-profile.webp 961w, https://www.datagaps.com/wp-content/uploads/data-profile-300x186.webp 300w, https://www.datagaps.com/wp-content/uploads/data-profile-768x477.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-0f838c9 elementor-widget elementor-widget-spacer" data-id="0f838c9" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-d0db329 elementor-widget__width-inherit elementor-widget elementor-widget-image" data-id="d0db329" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="284" src="https://www.datagaps.com/wp-content/uploads/data-rules-1024x455.webp" class="attachment-large size-large wp-image-6413" alt="data-rules" srcset="https://www.datagaps.com/wp-content/uploads/data-rules-1024x455.webp 1024w, https://www.datagaps.com/wp-content/uploads/data-rules-300x133.webp 300w, https://www.datagaps.com/wp-content/uploads/data-rules-768x342.webp 768w, https://www.datagaps.com/wp-content/uploads/data-rules.webp 1493w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-214e120 elementor-widget elementor-widget-spacer" data-id="214e120" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-d9be651 elementor-widget elementor-widget-text-editor" data-id="d9be651" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data Rules component</p> </div>
</div>
<div class="elementor-element elementor-element-a170b13 elementor-widget elementor-widget-heading" data-id="a170b13" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Step 3: Copy data to Snowflake</h5> </div>
</div>
<div class="elementor-element elementor-element-c82ee25 elementor-widget elementor-widget-text-editor" data-id="c82ee25" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Assuming that the Snowflake tables have been created, the last step is to copy the data to the snowflake. Use the VALIDATE function to validate the data files and identify any errors. DataFlow can be used to compare the data between the Staging Zone (S3) files and Snowflake after the load.</p> </div>
</div>
<div class="elementor-element elementor-element-31791e9 elementor-widget elementor-widget-text-editor" data-id="31791e9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>– Compare table to file row counts<br />– Compare data encoding<br />– Compare data completeness<br />– Compare data values<br />– End-to-end data validation (Legacy data warehouse to Snowflake)</p> </div>
</div>
<div class="elementor-element elementor-element-6b72073 elementor-widget elementor-widget-text-editor" data-id="6b72073" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataFlow can be used to perform end-to-end <a href="https://www.datagaps.com/blog/how-to-perform-continuous-data-validation/">Data Validation</a> in a single test as shown to the right. A single DataFlow can be used to compare data between legacy data warehouse and S3 as well as legacy data warehouse and Snowflake.</p> </div>
</div>
<div class="elementor-element elementor-element-1cce3be elementor-widget__width-inherit elementor-widget elementor-widget-image" data-id="1cce3be" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="633" height="338" src="https://www.datagaps.com/wp-content/uploads/End-to-end-data-validation.webp" class="attachment-large size-large wp-image-6423" alt="End-to-end-data-validation" srcset="https://www.datagaps.com/wp-content/uploads/End-to-end-data-validation.webp 633w, https://www.datagaps.com/wp-content/uploads/End-to-end-data-validation-300x160.webp 300w" sizes="(max-width: 633px) 100vw, 633px" /> </div>
</div>
<div class="elementor-element elementor-element-e9b99cc elementor-widget elementor-widget-spacer" data-id="e9b99cc" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-9db8d8c elementor-widget elementor-widget-text-editor" data-id="9db8d8c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>End-to-end test case</p> </div>
</div>
<div class="elementor-element elementor-element-7723177 elementor-widget elementor-widget-heading" data-id="7723177" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Step 4: Modify reports to Use Snowflake</h5> </div>
</div>
<div class="elementor-element elementor-element-b02f822 elementor-widget elementor-widget-text-editor" data-id="b02f822" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>While snowflake provides JDBC/ODBC drivers and supports most of the commonly used SQL functions, there are going to be some differences between the way reports are developed and executed in the legacy <a href="https://www.datagaps.com/services/data-warehouse-services/">Data Warehouse</a> and Snowflake. Once these changes are made, thorough testing needs to be performed between the reports using the legacy data warehouse and the equivalent reports using Snowflake.</p> </div>
</div>
<div class="elementor-element elementor-element-e9b85a9 elementor-widget elementor-widget-text-editor" data-id="e9b85a9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>– Compare report data<br />– Compare report layout<br />– Compare report performance<br />– Stress test reports in the new environments by simulating concurrent user loads<br />– Compare security</p> </div>
</div>
<div class="elementor-element elementor-element-0e76ec0 elementor-widget elementor-widget-text-editor" data-id="0e76ec0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Datagaps BI Validator is a no-code <a href="https://www.datagaps.com/bi-testing-tools/bi-validator/">BI Testing Tool</a> that can help automate all these tests for the supported BI tools.</p> </div>
</div>
<div class="elementor-element elementor-element-0c20832 elementor-widget elementor-widget-text-editor" data-id="0c20832" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Try BI Validator free for 14 days for your Snowflake BI testing needs – <a href="https://www.datagaps.com/bi-testing-tools/bi-validator-download/">Download Now</a></p> </div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/snowflake-data-migration-testing/">Data Validation for Snowflake</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Datagaps Attains Snowflake Select Technology Partner Status</title>
<link>https://www.datagaps.com/blog/datagaps-attains-snowflake-select-technology-partner-status/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 08 Feb 2022 19:52:46 +0000</pubDate>
<category><![CDATA[BI Testing]]></category>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=3955</guid>
<description><![CDATA[<p>Datagaps Inc. announces a new partnership with Snowflake, the Data Cloud company. Datagaps attains Snowflake Select Technology Partner in the space of Data and Business Intelligence Validation to help organizations validate their migration to Snowflake’s Data Cloud.</p>
<p>The post <a href="https://www.datagaps.com/blog/datagaps-attains-snowflake-select-technology-partner-status/">Datagaps Attains Snowflake Select Technology Partner Status</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="3955" class="elementor elementor-3955" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-2087d81 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="2087d81" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-5f57b97" data-id="5f57b97" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cd8e676 elementor-widget elementor-widget-text-editor" data-id="cd8e676" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a title="Datagaps" href="https://www.datagaps.com/" target="_blank" rel="noopener"><span style="text-decoration: underline; color: #1967d2;">Datagaps Inc</span>.</a> announces a new partnership with <a href="https://www.snowflake.com/en/"><span style="text-decoration: underline;"><span style="color: #1967d2; text-decoration: underline;">Snowflake</span></span></a>, the Data Cloud company. Datagaps attains Snowflake Select Technology Partner in the space of Data and Business Intelligence Validation to help organizations validate their migration to Snowflake’s Data Cloud.</p> </div>
</div>
<div class="elementor-element elementor-element-88bd883 elementor-widget elementor-widget-text-editor" data-id="88bd883" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Snowflake delivers the Data Cloud, where thousands of organizations have seamless access to explore, share, and unlock the true value of their data.</p> </div>
</div>
<div class="elementor-element elementor-element-0150e08 elementor-widget elementor-widget-text-editor" data-id="0150e08" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Datagaps as a Select Technology Partner helps organizations validate their legacy migration to Snowflake’s platform. Datagaps ETL Validation helps ensure that the ETL processes are running correctly when the new data structures are pointed to. The Data Migration Wizard provides a quick method for validating that hundreds of legacy tables moved to the cloud correctly. Finally, as the BI applications need to be redirected to the cloud, BI Validator includes regression testing, performance testing, stress testing, and old vs new GUI comparisons to test the BI migration.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-f0d890f elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="f0d890f" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-01e7abe" data-id="01e7abe" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-c9754a7 elementor-widget elementor-widget-image" data-id="c9754a7" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="523" height="477" src="https://www.datagaps.com/wp-content/uploads/Naren_headshot-01-1.webp" class="attachment-large size-large wp-image-6456" alt="Naren_headshot-01" srcset="https://www.datagaps.com/wp-content/uploads/Naren_headshot-01-1.webp 523w, https://www.datagaps.com/wp-content/uploads/Naren_headshot-01-1-300x274.webp 300w" sizes="(max-width: 523px) 100vw, 523px" /> </div>
</div>
<div class="elementor-element elementor-element-bf73fce elementor-widget elementor-widget-text-editor" data-id="bf73fce" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="et_pb_module_header">Naren <br />Yalamanchilli</p> </div>
</div>
<div class="elementor-element elementor-element-64332e2 elementor-widget elementor-widget-text-editor" data-id="64332e2" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="et_pb_module_header">CEO, Datagaps Inc.</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-96bb5c9" data-id="96bb5c9" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-2d8ed7b elementor-view-framed elementor-widget__width-initial elementor-absolute elementor-shape-circle elementor-widget elementor-widget-icon" data-id="2d8ed7b" data-element_type="widget" data-e-type="widget" data-settings="{"_position":"absolute"}" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="icon icon-double-quote"></i> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-d025ded elementor-widget elementor-widget-text-editor" data-id="d025ded" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><em><i>Snowflake’s Platform</i><i> presents key advantages over traditional on-premises data solutions. However, migrating to Snowflake can requires careful planning and execution. Datagaps helps simplify this process by automating the data migration testing and analytics regression testing. Using our validation, customers have successfully validated billions of records and thousands of reports. We are looking forward to helping more Snowflake customers with this partnership</i></em></p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-e6d1fa3 elementor-widget elementor-widget-text-editor" data-id="e6d1fa3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Moving data freely from one solution to another, and then to all your business users for truly democratized analytics, is not only vital to your organization but also one of your biggest challenges. Snowflake’s platform natively connects to dozens of other solutions, turning data into an asset you can easily derive valuable insights from without the cost, risk, and headache of trying to connect disparate, legacy systems.</p> </div>
</div>
<div class="elementor-element elementor-element-c56cede elementor-widget elementor-widget-text-editor" data-id="c56cede" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>With data today being the core of every business, it is essential that enterprises work with the right technology experts to get their outputs and analytics right. Datagaps data validation solutions, be it ETL Validator, BI Validator, or the DataOps Suite- (A complete suite for data validation and data quality with test data manager), have been very effective in achieving set data needs with all our customers.</p> </div>
</div>
<div class="elementor-element elementor-element-ae850d1 elementor-widget elementor-widget-spacer" data-id="ae850d1" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-dcc18bf elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="dcc18bf" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-11857a1" data-id="11857a1" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-e6d3e36 elementor-widget__width-initial elementor-absolute elementor-view-framed elementor-shape-circle elementor-widget elementor-widget-icon" data-id="e6d3e36" data-element_type="widget" data-e-type="widget" data-settings="{"_position":"absolute"}" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="icon icon-double-quote"></i> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-4c04770 elementor-widget elementor-widget-text-editor" data-id="4c04770" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>We have seen a significant increase in inquiries regarding our ability to help validate the data migration process to Snowflake. People have turned to us to help validate the data migration, new vs. old ETL/ELT process data validation and on the backend, validate the data analytics old vs. new now pointing to Snowflake. We are excited about the opportunity to work with Snowflake’s clients and are thrilled to have achieved Select Partner Status with Snowflake.</p> </div>
</div>
<div class="elementor-element elementor-element-b425edd elementor-widget elementor-widget-text-editor" data-id="b425edd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p class="et_pb_module_header"><strong>David Small- VP Sales and Marketing, Datagaps Inc.</strong></p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-f9430e0 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="f9430e0" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/datagaps-attains-snowflake-select-technology-partner-status/">Datagaps Attains Snowflake Select Technology Partner Status</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Datagaps Data Validation and Migration To Snowflake</title>
<link>https://www.datagaps.com/blog/data-validation-in-snowflake-data-migration/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Wed, 19 Jan 2022 12:31:22 +0000</pubDate>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7286</guid>
<description><![CDATA[<p>This article describes ways in which we can test the data movement between Snowflake and on-premises data and between instances of Snowflake itself. We will also provide some benchmarks for doing comparison testing for large volumes of data moved into Snowflake.</p>
<p>The post <a href="https://www.datagaps.com/blog/data-validation-in-snowflake-data-migration/">Datagaps Data Validation and Migration To Snowflake</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7286" class="elementor elementor-7286" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-be777ec elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="be777ec" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-8fc52a4" data-id="8fc52a4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-deaa6e9 elementor-widget elementor-widget-heading" data-id="deaa6e9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Data validation in snowflake</h2> </div>
</div>
<div class="elementor-element elementor-element-42e8dbe elementor-widget elementor-widget-text-editor" data-id="42e8dbe" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>This article discusses why and how to use both together, and dives into the challenges of Bulk <span style="text-decoration: underline;"><span style="color: #1967d2; text-decoration: underline;"><a style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/cloud-data-test-automation/snowflake/" target="_blank" rel="noopener">Data Migration to Snowflake.</a></span></span></p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-b696c59 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="b696c59" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0eca593" data-id="0eca593" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-56519fd elementor-widget elementor-widget-heading" data-id="56519fd" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Why and How?</h5> </div>
</div>
<div class="elementor-element elementor-element-e15c457 elementor-widget elementor-widget-text-editor" data-id="e15c457" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
People are rapidly adopting cloud architectures for Data Warehouses and Machine Learning projects due to the economies of scale in the cloud. One obstacle in achieving rapid success is the data and data types inconsistencies between on-premise structures and the modern data stacks in the cloud. </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-07ab3d6" data-id="07ab3d6" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4926655 elementor-widget elementor-widget-video" data-id="4926655" data-element_type="widget" data-e-type="widget" data-settings="{"video_type":"hosted","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="e-hosted-video elementor-wrapper elementor-open-inline">
<video class="elementor-video" src="https://datagapscustomerdemos.s3.amazonaws.com/DataOps_MigrationWizard.mp4" controls="" preload="metadata" controlsList="nodownload"></video>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-c76830f elementor-widget elementor-widget-text-editor" data-id="c76830f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
When you migrate vast amounts of data to the cloud, the opportunity to introduce mistakes is multiplied due to these reasons and others. The earlier you catch the issues, the less costly it is to resolve the discrepancies. </div>
</div>
<div class="elementor-element elementor-element-9f1dcd3 elementor-widget elementor-widget-text-editor" data-id="9f1dcd3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>This is where </strong><span style="text-decoration: underline; color: #1967d2;"><a style="color: #1967d2; text-decoration: underline;" href="https://www.datagaps.com/" target="_blank" rel="noopener">Datagaps</a></span><strong> come in to play.</strong></p> </div>
</div>
<div class="elementor-element elementor-element-bf22b32 elementor-widget elementor-widget-spacer" data-id="bf22b32" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-f1a6cdd elementor-section-content-top bw-ac quote-box elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="f1a6cdd" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-b7124dd" data-id="b7124dd" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0f81d89 elementor-view-default elementor-widget elementor-widget-icon" data-id="0f81d89" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="icon icon-double-quote"></i> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-5fb2cbc elementor-widget elementor-widget-text-editor" data-id="5fb2cbc" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/wp-content/uploads/2022/01/database-and-Snowflake.svg"><br /><img loading="lazy" decoding="async" class="attachment-thumbnail size-thumbnail wp-image-6747" src="https://www.datagaps.com/wp-content/uploads/database-and-Snowflake.svg" alt="database-and-Snowflake" width="129" height="150" /> </a><br />We provide the ability to test millions & even billions of records between source and targets structures of different types such as your on premises database and Snowflake.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-86a47c9 elementor-widget elementor-widget-text-editor" data-id="86a47c9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
This article describes ways in which we can test the data movement between Snowflake and on-premises data and between instances of Snowflake itself. We will also provide some benchmarks for doing comparison testing for large volumes of data moved into Snowflake. Another benefit of the Datagaps approach is, as additional data is moved into the Snowflake structure from other sources, Datagaps provides the ability to monitor the quality of your data structures by continually providing an up to date scoring so that you can determine when data is becoming corrupted. </div>
</div>
<div class="elementor-element elementor-element-765422c elementor-widget elementor-widget-heading" data-id="765422c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Benefits of using Datagaps to test data movement into Snowflake</h2> </div>
</div>
<div class="elementor-element elementor-element-58e202c elementor-widget elementor-widget-text-editor" data-id="58e202c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>We recently sat down with one of our clients that uses our DataFlow product for testing the data migration from on-Prem SQL Server to Snowflake in the cloud running in <span style="text-decoration: underline; color: #1967d2;"><a style="color: #1967d2;" href="https://aws.amazon.com/" target="_blank" rel="noopener">AWS.</a></span></p> </div>
</div>
<div class="elementor-element elementor-element-161237a elementor-widget elementor-widget-image" data-id="161237a" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="464" height="53" src="https://www.datagaps.com/wp-content/uploads/Datagaps-Snowflake_1.svg" class="attachment-full size-full wp-image-6812" alt="Datagaps-Snowflake" /> </div>
</div>
<div class="elementor-element elementor-element-aac78ca elementor-widget elementor-widget-spacer" data-id="aac78ca" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-bb16fa5 elementor-widget elementor-widget-heading" data-id="bb16fa5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Implementation</h5> </div>
</div>
<div class="elementor-element elementor-element-153c094 elementor-widget elementor-widget-text-editor" data-id="153c094" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>They started the initial migration by performing bulk loads from 400 SQL Server tables to Snowflake with minimal transformations. This was stage 0, where they could perform source and target data comparison for over 500 million rows of data per table. Making use of the Data Migration wizard, the client was able to generate comparison tests for 400 tables in just a few hours. Even though there were few changes at this stage, they still encountered errors that were surfaced by our<br />DataFlow product.</p> </div>
</div>
<div class="elementor-element elementor-element-4d1c640 elementor-widget elementor-widget-text-editor" data-id="4d1c640" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Examples:</strong> Issues include numeric precision differences, null value inconsistencies, truncation issues and character interpolations. </div>
</div>
<div class="elementor-element elementor-element-285f521 elementor-widget elementor-widget-image" data-id="285f521" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="522" height="174" src="https://www.datagaps.com/wp-content/uploads/Datagaps-Snowflake.svg" class="attachment-full size-full wp-image-6814" alt="Datagaps-Snowflake" /> </div>
</div>
<div class="elementor-element elementor-element-67513ee elementor-widget elementor-widget-text-editor" data-id="67513ee" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Next, they began to perform incremental new data migrations where they continued to find similar issues that had to be corrected. As this continued, they wanted to transition from this incremental new data migration from SQL Server to loading the new data directly into Snowflake to reap the benefits stated earlier. To accomplish this, their initial ETL processes needed to be migrated to an ELT process aimed at Snowflake.</p> </div>
</div>
<div class="elementor-element elementor-element-e9d5e95 elementor-widget elementor-widget-text-editor" data-id="e9d5e95" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataFlow was used once again to check the accuracy between the two systems once the new processes were in place. The validations exposed issues in the new ELT process through several iterations until the transformation were in sync. After a short period of testing, they could cut over to the new system and deprecate the SQL server environment. Now DataFlow continued to validate the incremental data as it was moved into Snowflake, finding issues earlier in the cycle where they are less costly to fix in time and lost credibility.</p> </div>
</div>
<div class="elementor-element elementor-element-7847517 elementor-widget elementor-widget-heading" data-id="7847517" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">How it makes a difference?</h5> </div>
</div>
<div class="elementor-element elementor-element-364be8e elementor-widget elementor-widget-text-editor" data-id="364be8e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Many of our clients take this one step further by testing their BI tools against the old warehouse and the new Snowflake implementation. They do regression, performance, and stress testing using our BI Validator tool to compare the old with the new. They can find differences in the look and feel in the output generation of the reports and dashboards. Differences are exposed between the report queries when compared to a database query. Often this is the last task necessary to validate the migration process. </div>
</div>
<div class="elementor-element elementor-element-0986aa8 elementor-widget elementor-widget-spacer" data-id="0986aa8" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-63da614 elementor-align-center promo-button elementor-invisible elementor-widget elementor-widget-button" data-id="63da614" data-element_type="widget" data-e-type="widget" data-settings="{"_animation":"fadeInLeft"}" data-widget_type="button.default">
<div class="elementor-widget-container">
<div class="elementor-button-wrapper">
<a class="elementor-button elementor-button-link elementor-size-sm" href="#">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-icon">
<i aria-hidden="true" class="fas fa-download"></i> </span>
<span class="elementor-button-text">Request a Demo</span>
</span>
</a>
</div>
</div>
</div>
<div class="elementor-element elementor-element-79f9cc4 elementor-widget elementor-widget-spacer" data-id="79f9cc4" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-93243e1 elementor-widget elementor-widget-text-editor" data-id="93243e1" data-element_type="widget" data-e-type="widget" id="more" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Making use of the inbuilt cluster integration with AWS EMR in the DataOps suite, the client was able to automatically scale the EMR cluster on demand to over 30 nodes depending on the data volumes being compared and scale down once the testing has been completed. This capability helped reduce the cost of the testing while still achieving high performance when comparing tables of size <strong>500 Million Records</strong> </div>
</div>
<div class="elementor-element elementor-element-02284ef elementor-widget elementor-widget-spacer" data-id="02284ef" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6c1758c elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6c1758c" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-2a1dfc8" data-id="2a1dfc8" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-18f1145 elementor-widget elementor-widget-premium-addon-progressbar" data-id="18f1145" data-element_type="widget" data-e-type="widget" data-widget_type="premium-addon-progressbar.default">
<div class="elementor-widget-container">
<div class="premium-progressbar-container" data-settings="{"progress_length":"60","speed":1000,"type":"circle","mScroll":""}">
<div class="premium-progressbar-circle-wrap">
<div class="premium-progressbar-circle-base"></div>
<div class="premium-progressbar-circle">
<div class="premium-progressbar-circle-left"></div>
<div class="premium-progressbar-circle-right"></div>
</div>
<div class="premium-progressbar-circle-content">
<p class="premium-progressbar-left-label">
</p>
<p class="premium-progressbar-right-label">0%</p>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e6025e9 elementor-widget elementor-widget-heading" data-id="e6025e9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h6 class="elementor-heading-title elementor-size-default">Reduction in Testing <br> Time by</h6> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-b4a575f" data-id="b4a575f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-24aed22 elementor-widget elementor-widget-premium-addon-progressbar" data-id="24aed22" data-element_type="widget" data-e-type="widget" data-widget_type="premium-addon-progressbar.default">
<div class="elementor-widget-container">
<div class="premium-progressbar-container" data-settings="{"progress_length":"50","speed":1000,"type":"circle","mScroll":""}">
<div class="premium-progressbar-circle-wrap">
<div class="premium-progressbar-circle-base"></div>
<div class="premium-progressbar-circle">
<div class="premium-progressbar-circle-left"></div>
<div class="premium-progressbar-circle-right"></div>
</div>
<div class="premium-progressbar-circle-content">
<p class="premium-progressbar-left-label">
</p>
<p class="premium-progressbar-right-label">0%</p>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e6058cc elementor-widget elementor-widget-heading" data-id="e6058cc" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h6 class="elementor-heading-title elementor-size-default">Improved Testing ROI by</h6> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-9344e70" data-id="9344e70" data-element_type="column" data-e-type="column" data-settings="{"background_background":"classic"}">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-68b24ab elementor-widget__width-initial elementor-absolute elementor-view-default elementor-widget elementor-widget-icon" data-id="68b24ab" data-element_type="widget" data-e-type="widget" data-settings="{"_position":"absolute"}" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="icon icon-double-quote"></i> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-7299e88 elementor-widget elementor-widget-heading" data-id="7299e88" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Amount of data tested increased from manual testing of <strong>10,000 sample records</strong> to complete testing of <strong>500 M</strong>.</h3> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-d796304 elementor-widget elementor-widget-spacer" data-id="d796304" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-7174f43 elementor-widget elementor-widget-spacer" data-id="7174f43" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-3494ee4 elementor-widget elementor-widget-image" data-id="3494ee4" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="595" height="96" src="https://www.datagaps.com/wp-content/uploads/Datagaps-Snowflake_Banner.svg" class="attachment-full size-full wp-image-6856" alt="Datagaps-Snowflake_Banner" /> </div>
</div>
<div class="elementor-element elementor-element-7f555a5 elementor-widget elementor-widget-spacer" data-id="7f555a5" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-4c7a1af elementor-widget elementor-widget-heading" data-id="4c7a1af" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-7f38d40 elementor-widget elementor-widget-text-editor" data-id="7f38d40" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In conclusion, the goals of the migration project of agility, cost savings and performance improvements were achieved.</p><p>They also realized these benefits months earlier as a result of the improvement in the migration process due to the impact of the DataFlow products contribution in an estimated 50% test cycle reduction.</p><p>One of our clients reports comparing a file against a <strong>Snowflake</strong> instance with</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-ce4b0c6 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="ce4b0c6" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-25 elementor-inner-column elementor-element elementor-element-b7a2be1" data-id="b7a2be1" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0ca84ee elementor-widget elementor-widget-counter" data-id="0ca84ee" data-element_type="widget" data-e-type="widget" data-widget_type="counter.default">
<div class="elementor-widget-container">
<div class="elementor-counter">
<div class="elementor-counter-title">Billion Records</div> <div class="elementor-counter-number-wrapper">
<span class="elementor-counter-number-prefix">0</span>
<span class="elementor-counter-number" data-duration="2000" data-to-value="1" data-from-value="0" data-delimiter=",">0</span>
<span class="elementor-counter-number-suffix"></span>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-25 elementor-inner-column elementor-element elementor-element-ef86f0a" data-id="ef86f0a" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-325db7a elementor-widget elementor-widget-counter" data-id="325db7a" data-element_type="widget" data-e-type="widget" data-widget_type="counter.default">
<div class="elementor-widget-container">
<div class="elementor-counter">
<div class="elementor-counter-title">Columns</div> <div class="elementor-counter-number-wrapper">
<span class="elementor-counter-number-prefix"></span>
<span class="elementor-counter-number" data-duration="2000" data-to-value="23" data-from-value="0" data-delimiter=",">0</span>
<span class="elementor-counter-number-suffix"></span>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-25 elementor-inner-column elementor-element elementor-element-c33b403" data-id="c33b403" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-2461f1a elementor-widget elementor-widget-counter" data-id="2461f1a" data-element_type="widget" data-e-type="widget" data-widget_type="counter.default">
<div class="elementor-widget-container">
<div class="elementor-counter">
<div class="elementor-counter-title">Node EMR Cluster</div> <div class="elementor-counter-number-wrapper">
<span class="elementor-counter-number-prefix"></span>
<span class="elementor-counter-number" data-duration="2000" data-to-value="10" data-from-value="0" data-delimiter=",">0</span>
<span class="elementor-counter-number-suffix"></span>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-25 elementor-inner-column elementor-element elementor-element-ee098b8" data-id="ee098b8" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8e57fd7 elementor-widget elementor-widget-counter" data-id="8e57fd7" data-element_type="widget" data-e-type="widget" data-widget_type="counter.default">
<div class="elementor-widget-container">
<div class="elementor-counter">
<div class="elementor-counter-title">Hours</div> <div class="elementor-counter-number-wrapper">
<span class="elementor-counter-number-prefix"></span>
<span class="elementor-counter-number" data-duration="2000" data-to-value="1.5" data-from-value="0" data-delimiter=".">0</span>
<span class="elementor-counter-number-suffix"></span>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-a3b0f28 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="a3b0f28" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/data-validation-in-snowflake-data-migration/">Datagaps Data Validation and Migration To Snowflake</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p>
]]></content:encoded>
</item>
</channel>
</rss>