<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
>
<channel>
<title>Cloud Data Migration Archives - Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</title>
<atom:link href="https://www.datagaps.com/blog/category/cloud-data-migration/feed/" rel="self" type="application/rss+xml" />
<link>https://www.datagaps.com/blog/category/cloud-data-migration/</link>
<description></description>
<lastBuildDate>Mon, 19 Jan 2026 07:20:57 +0000</lastBuildDate>
<language>en-US</language>
<sy:updatePeriod>
hourly </sy:updatePeriod>
<sy:updateFrequency>
1 </sy:updateFrequency>
<generator>https://wordpress.org/?v=6.9.4</generator>
<image>
<url>https://www.datagaps.com/wp-content/uploads/Datagaps-India-Favicon-Lite-theme-150x150.jpg</url>
<title>Cloud Data Migration Archives - Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</title>
<link>https://www.datagaps.com/blog/category/cloud-data-migration/</link>
<width>32</width>
<height>32</height>
</image>
<item>
<title>Top 3 ETL Testing Tools</title>
<link>https://www.datagaps.com/blog/top-3-etl-testing-tools/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Thu, 05 Jun 2025 17:38:50 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7034</guid>
<description><![CDATA[<p>ETL Testing refers to the testing, validation, and analysis of the Extraction, Transformation, and Loading Processes that are part of ETL and ELT Pipelines. As ETL testing refers to “Data-in-Motion” Testing, the unit test architecture and principles slightly differ from “Data-at-Rest” Testing (Warehouse/DB Validation).</p>
<p>The post <a href="https://www.datagaps.com/blog/top-3-etl-testing-tools/">Top 3 ETL Testing Tools</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7034" class="elementor elementor-7034" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-95ac9c1 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="95ac9c1" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-extended">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e6bbc9f" data-id="e6bbc9f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-b8c33c8 elementor-widget elementor-widget-heading" data-id="b8c33c8" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">What Is ETL Testing Tools?</h2> </div>
</div>
<div class="elementor-element elementor-element-f29b3ae elementor-widget elementor-widget-text-editor" data-id="f29b3ae" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL Testing refers to the testing, validation, and analysis of the Extraction, Transformation, and Loading Processes that are part of ETL and ELT Pipelines. As <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> refers to “Data-in-Motion” Testing, the unit test architecture and principles slightly differ from “Data-at-Rest” Testing (Warehouse/DB Validation).</p><p>For any Top ETL Testing Tool, a typical task list of an ETL Testing Model must include:</p><ul class="custom-list"><li>Data Model Review</li><li>Source Data Testing</li><li>Post-Ingestion Validation</li><li>Post-Transform Validation</li><li>Aggregation Analysis</li><li>Data Compare between Source and Target</li><li>Data Quality and Accuracy Testing in Target</li><li>Data Integrity Examination</li><li>ETL Operational Update Validation</li><li><a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL Performance Testing</a></li></ul><p>As ETL Pipelines contain most of the transformations, relations, and aggregations that will be performed, the majority of errors occur in these sets. Even with a static Database source, as functions get updated and changed the errors can creep up even in a stable ETL pipeline.</p><p>Also read: <span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/blog/etl-validator-for-data-migration-testing/">ETL Validator for Data Migration Testing</a></span></p> </div>
</div>
<div class="elementor-element elementor-element-5a2c102 elementor-widget elementor-widget-heading" data-id="5a2c102" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Why Is ETL Testing Important?</h2> </div>
</div>
<div class="elementor-element elementor-element-41f58dd elementor-widget elementor-widget-text-editor" data-id="41f58dd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL testing is important for several reasons:</p><ul><li><strong>First,</strong> it ensures the integrity and reliability of the data being used in a data warehousing or business intelligence system. By verifying the accuracy and completeness of the data, ETL testing helps to ensure that the decisions made based on that data are correct and accurate.</li><li><strong>Second,</strong> ETL testing helps to identify and resolve any issues or errors in the ETL process. This can prevent data loss and improve the overall performance of the system. For example, if an ETL test detects that certain data is missing or incorrect, the issue can be quickly addressed and corrected, which can improve the quality of the data and the reliability of the system.</li><li><strong>Third,</strong> ETL testing can help to ensure compliance with industry standards and regulations. Many industries have specific requirements for the handling and processing of data, and ETL testing can help to ensure that the data being used in a data warehousing or business intelligence system meets these requirements. This can prevent fines and penalties for non-compliance, and can also help to protect the reputation of the organization.</li></ul><p>Overall, ETL testing is a critical step in the data warehousing and business intelligence process, and it is essential for ensuring the accuracy and reliability of the data being used in these systems. Hence we should more cautious in picking the right ETL Testing Tool.</p> </div>
</div>
<div class="elementor-element elementor-element-18ad620 elementor-widget elementor-widget-heading" data-id="18ad620" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-medium">Here are some Top ETL Testing Tools available in the market.
</h2> </div>
</div>
<div class="elementor-element elementor-element-aca931e elementor-widget elementor-widget-heading" data-id="aca931e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">#1 ETL Validator</h3> </div>
</div>
<div class="elementor-element elementor-element-82c79e6 elementor-widget elementor-widget-text-editor" data-id="82c79e6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong><span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/etl-testing-tools/etl-validator/">DataGaps ETL Validator</a> </span></strong>stands on top when it comes to ETL Testing Automation. This is now part of DataOps Suite.</p> </div>
</div>
<div class="elementor-element elementor-element-2f6d8a0 elementor-widget elementor-widget-image" data-id="2f6d8a0" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img fetchpriority="high" decoding="async" width="640" height="299" src="https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-1024x479.webp" class="attachment-large size-large wp-image-5333" alt="ETL-Validator-01" srcset="https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-1024x479.webp 1024w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-300x140.webp 300w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-768x359.webp 768w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-1536x718.webp 1536w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01.webp 1920w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-e74ffa9 elementor-widget elementor-widget-text-editor" data-id="e74ffa9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The DataGaps ETL Validator is a tool that helps organizations ensure the quality and integrity of their data as it is transferred from one system to another through the process of Extract, Transform, and Load (ETL). The ETL Validator checks the data against a set of pre-defined rules and constraints and identifies any errors or inconsistencies that may be present. This can help organizations avoid problems such as incorrect data being loaded into their systems, or data being lost or corrupted during the ETL process.</p><p>One of the key features of the DataGaps ETL Validator is its ability to handle large amounts of data quickly and efficiently. This is important because ETL processes often involve moving large volumes of data from multiple sources, and the Validator can help organizations ensure that their data is transferred accurately and without delays.</p><p>Another important feature of the DataGaps ETL Validator is its ability to identify and highlight any errors or inconsistencies in the data. This can help organizations quickly identify and fix any issues, and ensure that their data is accurate and complete. The Validator also provides detailed reports and logs, which can be used to track the progress of the ETL process and troubleshoot any problems that may arise.</p> </div>
</div>
<div class="elementor-element elementor-element-87fdce9 elementor-widget elementor-widget-image" data-id="87fdce9" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img decoding="async" width="640" height="300" src="https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-1024x480.webp" class="attachment-large size-large wp-image-5334" alt="ETL-Validator-02" srcset="https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-1024x480.webp 1024w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-300x141.webp 300w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-768x360.webp 768w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-1536x720.webp 1536w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02.webp 1920w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-e331430 elementor-widget elementor-widget-text-editor" data-id="e331430" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, the DataGaps ETL Validator is a valuable and Top ETL Testing tool for organizations that need to ensure the quality and integrity of their data as it is transferred from one system to another. By providing fast, efficient, and accurate data validation, the Validator can help organizations avoid costly errors and improve the reliability and effectiveness of their ETL processes.</p> </div>
</div>
<div class="elementor-element elementor-element-142a5d9 elementor-widget elementor-widget-html" data-id="142a5d9" data-element_type="widget" data-e-type="widget" data-widget_type="html.default">
<div class="elementor-widget-container">
<div class="trigger-video" data-video-url="https://www.youtube.com/watch?v=GmgCYKGZn4I" style="position: relative; cursor: pointer;">
<img decoding="async" src="https://www.datagaps.com/wp-content/uploads/Top-3-ETL-Testing-Tools-Comparison.jpg" alt="Top 3 ETL Testing Tools Comparison" style="width: 100%; height: auto;border-radius:10px">
<!-- SVG Play Icon -->
<!-- Smaller SVG Play Icon -->
<div style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;">
<svg width="60px" viewBox="0 0 68 48" xmlns="http://www.w3.org/2000/svg">
<path class="ytp-large-play-button-bg"
d="M66.52,7.74c-0.78-2.93-2.49-5.41-5.42-6.19C55.79,.13,34,0,34,0S12.21,.13,6.9,1.55
C3.97,2.33,2.27,4.81,1.48,7.74C0.06,13.05,0,24,0,24s0.06,10.95,1.48,16.26c0.78,2.93,2.49,5.41,5.42,6.19
C12.21,47.87,34,48,34,48s21.79-0.13,27.1-1.55c2.93-0.78,4.64-3.26,5.42-6.19C67.94,34.95,68,24,68,24S67.94,13.05,66.52,7.74z"
fill="#f03" />
<path d="M 45,24 27,14 27,34" fill="#fff" />
</svg>
</div>
</div>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "VideoObject",
"name": "Top 3 ETL Testing Tools Comparison",
"description": "we break down everything you need to know about ETL Testing Tools, how they work, and which tools lead the market in 2026.",
"thumbnailUrl": "https://www.datagaps.com/wp-content/uploads/Top-3-ETL-Testing-Tools-Comparison.jpg",
"uploadDate": "2025-10-31T12:00:00Z",
"duration": "PT5M59S",
"publisher": {
"@type": "Organization",
"name": "Datagaps",
"logo": {
"@type": "ImageObject",
"url": "https://www.datagaps.com/wp-content/uploads/datagaps-logo.svg"
}
},
"contentUrl": "https://www.youtube.com/watch?v=GmgCYKGZn4I",
"embedUrl": "https://www.youtube.com/embed/GmgCYKGZn4I",
"interactionStatistic": {
"@type": "InteractionCounter",
"interactionType": { "@type": "http://schema.org/WatchAction" },
"userInteractionCount": "10"
},
"regionsAllowed": ["US", "CA", "IN","GB","AU","DE","FR","IT","ES","JP","CN","RU"]
}
</script> </div>
</div>
<div class="elementor-element elementor-element-d86abb7 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="d86abb7" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-2f31c82 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="2f31c82" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-4d3d257" data-id="4d3d257" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-11c46ae elementor-widget elementor-widget-text-editor" data-id="11c46ae" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/etl-validator-trial-request/">ETL Validator – Free Trial</a></span></p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-e20f0d7" data-id="e20f0d7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-f5ebf40 elementor-widget elementor-widget-text-editor" data-id="f5ebf40" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/blog/how-to-validate-etl-testing-checklist/">ETL Testing Tool Checklist</a></span></p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-b302254" data-id="b302254" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-74398c6 elementor-widget elementor-widget-text-editor" data-id="74398c6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #339966;"><a style="color: #339966;" href="https://www.youtube.com/watch?v=j4rAuW7I7Do" data-wplink-edit="true">ETL Validator – Webinar</a></span></p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-0055398 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="0055398" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-0ac692f elementor-widget elementor-widget-heading" data-id="0ac692f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">#2 QuerySurge</h3> </div>
</div>
<div class="elementor-element elementor-element-25cddc5 elementor-widget elementor-widget-text-editor" data-id="25cddc5" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>QuerySurge is a powerful ETL Testing tool designed to help businesses and organizations quickly and efficiently test and validate their data. With its intuitive interface and robust set of features, QuerySurge makes it easy to ensure that your data is accurate, complete, and ready for use.</p><p>One of the key features of QuerySurge ETL Testing Tool is its ability to automatically generate and execute test cases. This means that you can quickly and easily test your data without having to manually write and run individual test cases. QuerySurge also allows you to specify the criteria for each test, so you can tailor your tests to fit the specific needs of your organization.</p><p>Another important feature of QuerySurge is its ability to integrate with a wide range of data sources. This means that you can use QuerySurge to test data from a variety of sources, including databases, flat files, and even web services. This flexibility allows you to easily test data from multiple sources and ensure that your data is consistent and accurate across all of your systems.</p><p>In addition to its automation and data integration capabilities, QuerySurge also offers a number of other powerful features. For example, QuerySurge allows you to define and manage your test data sets, so you can easily reuse test data and maintain a consistent testing environment. QuerySurge also provides detailed reporting capabilities, so you can easily track the progress of your tests and identify any potential issues.</p> </div>
</div>
<div class="elementor-element elementor-element-3888c62 elementor-widget elementor-widget-image" data-id="3888c62" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img decoding="async" width="640" height="328" src="https://www.datagaps.com/wp-content/uploads/QuerySurge.webp" class="attachment-large size-large wp-image-5517" alt="QuerySurge" srcset="https://www.datagaps.com/wp-content/uploads/QuerySurge.webp 745w, https://www.datagaps.com/wp-content/uploads/QuerySurge-300x154.webp 300w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-007dd63 elementor-widget elementor-widget-text-editor" data-id="007dd63" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Overall, QuerySurge is a valuable tool for anyone looking to efficiently and effectively test and validate their data. With its powerful features and intuitive interface, QuerySurge makes it easy to ensure that your data is accurate and ready for use. </div>
</div>
<div class="elementor-element elementor-element-62f3f0b elementor-widget elementor-widget-heading" data-id="62f3f0b" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">#3 iCEDQ</h3> </div>
</div>
<div class="elementor-element elementor-element-a606638 elementor-widget elementor-widget-text-editor" data-id="a606638" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>iCEDQ is a powerful data quality management tool designed to help businesses and organizations ensure the accuracy and completeness of their data. With its intuitive interface and robust set of features, iCEDQ makes it easy to identify and correct data errors, ensuring that your data is clean and ready for use.</p>
<p>One of the key features of iCEDQ is its ability to automatically identify and flag potential data errors. Using a variety of algorithms and techniques, iCEDQ can quickly and accurately detect errors in your data, such as missing values, incorrect formatting, and inconsistencies. This allows you to quickly and easily identify areas where your data may be incorrect, so you can take action to fix the errors.</p>
In addition to its error detection capabilities, iCEDQ also offers a number of other powerful features. For example, iCEDQ allows you to define and manage your data quality rules, so you can easily ensure that your data meets the specific requirements of your organization. iCEDQ also provides detailed reporting capabilities, so you can track the progress of your data quality efforts and identify areas where you may need to take action. </div>
</div>
<div class="elementor-element elementor-element-c64dee0 elementor-widget elementor-widget-image" data-id="c64dee0" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="266" src="https://www.datagaps.com/wp-content/uploads/iCEDQ.webp" class="attachment-large size-large wp-image-5520" alt="iCEDQ" srcset="https://www.datagaps.com/wp-content/uploads/iCEDQ.webp 748w, https://www.datagaps.com/wp-content/uploads/iCEDQ-300x125.webp 300w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-d041cd6 elementor-widget elementor-widget-text-editor" data-id="d041cd6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, iCEDQ is a valuable tool for anyone looking to improve the quality of their data. With its powerful features and intuitive interface, iCEDQ makes it easy to identify and correct errors in your data, ensuring that it is accurate and reliable.</p> </div>
</div>
<div class="elementor-element elementor-element-f770149 elementor-widget elementor-widget-text-editor" data-id="f770149" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="text-decoration: underline;">Disclaimer</span>: The above-mentioned list is purely an outcome of the conversations and feedback received from various industry users in the ETL/Data Warehouse testing space. Any concerns or views can be shared at <a href="mailto:contact@datagaps.com">contact@datagaps.com</a></p> </div>
</div>
<div class="elementor-element elementor-element-fc6bada elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="fc6bada" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-11a47c6 e-flex e-con-boxed e-con e-parent" data-id="11a47c6" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-0b86605 e-con-full e-flex e-con e-child" data-id="0b86605" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-bbc97a9 elementor-widget elementor-widget-heading" data-id="bbc97a9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Try ETL Validator testing tool <span style="text-decoration: underline">Free for 14</span> days for your ETL Testing Automation needs. <a href="https://www.datagaps.com/etl-validator-trial-request/">Free Trial</a></h2> </div>
</div>
</div>
<div class="elementor-element elementor-element-f932ba0 e-con-full e-flex e-con e-child" data-id="f932ba0" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-ea2db0b elementor-align-right elementor-widget elementor-widget-button" data-id="ea2db0b" data-element_type="widget" data-e-type="widget" data-widget_type="button.default">
<div class="elementor-widget-container">
<div class="elementor-button-wrapper">
<a class="elementor-button elementor-button-link elementor-size-md" href="https://www.datagaps.com/request-demo/">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Request Demo</span>
</span>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/top-3-etl-testing-tools/">Top 3 ETL Testing Tools</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>The Impact of Automated Testing on Data Migration</title>
<link>https://www.datagaps.com/blog/future-impact-of-automated-testing-on-data-migration/</link>
<dc:creator><![CDATA[Eshaa Shah]]></dc:creator>
<pubDate>Mon, 25 Mar 2024 12:57:15 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Automated Data Migration Testing]]></category>
<guid isPermaLink="false">https://www.datagaps.com/?p=26602</guid>
<description><![CDATA[<p>Explore how to reshape the future of data migration, harness the full potential of accurate, reliable data migration testing with DataOps Suite.</p>
<p>The post <a href="https://www.datagaps.com/blog/future-impact-of-automated-testing-on-data-migration/">The Impact of Automated Testing on Data Migration</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="26602" class="elementor elementor-26602" data-elementor-post-type="post">
<div class="elementor-element elementor-element-ad2d90c e-flex e-con-boxed e-con e-parent" data-id="ad2d90c" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-2388682 elementor-widget elementor-widget-text-editor" data-id="2388682" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="none"><span style="color: #0000ff;"><a style="color: #0000ff;" href="https://en.wikipedia.org/wiki/Data_migration">Data migration</a></span> is a crucial aspect of modern business, allowing organizations to transition their data assets across various systems and platforms seamlessly. With cloud computing becoming increasingly popular, cloud migration is now a top priority for businesses looking to leverage cloud environments’ flexibility, scalability, and efficiency. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">Snowflake, a leading cloud data warehousing provider, is the go-to destination for enterprises seeking to optimize their data strategies through cloud migration. Migrating to the cloud can be tedious and complex, from ensuring data integrity to adapting to new data management paradigms. This is where automated testing comes in, providing efficient and reliable solutions for safeguarding data integrity, ensuring seamless transitions, and optimizing cloud capabilities.</span></p><p><span data-contrast="none">As businesses embrace digital transformation, the role of automated testing in crafting future-ready data ecosystems cannot be overstated. It paves the way for unlocking the full potential of data management strategies, emphasizing precision, speed, and strategic foresight.</span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">This blog explores the dynamic landscape of data migration and highlights the indispensable role of automation in building a resilient and forward-thinking data ecosystem</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-7fa3df8 e-flex e-con-boxed e-con e-parent" data-id="7fa3df8" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-3a3e80d elementor-widget elementor-widget-heading" data-id="3a3e80d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">The Importance of Data Migration in Business Strategy</h2> </div>
</div>
<div class="elementor-element elementor-element-84805bc elementor-widget elementor-widget-text-editor" data-id="84805bc" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="none">Automated data migration has transcended beyond mere operational requirements to becoming a cornerstone of strategic business innovation. It is the engine driving business growth, facilitating the seamless expansion of enterprises, and paving the way for a holistic digital transformation. This strategic process ensures that organizations remain agile, data-driven, and competitive in a market where the only constant is change. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">The stakes have never been higher for CIOs, CDOs, data engineers, and BI and data migration testing professionals. These key players are at the forefront of navigating complex data landscapes, ensuring data moves across systems and platforms efficiently and retains its integrity, accuracy, and relevance. The role of BI testing in this context is invaluable, offering a systematic approach to validating data quality, performance, and reliability across diverse BI environments and reporting tools. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">The strategic imperative of data migration lies in its ability to unlock actionable insights from vast data reserves, thereby informing critical decision-making processes and enhancing operational efficiencies. It ensures seamless data flow from legacy systems to modern, scalable platforms like cloud data warehouses, which support advanced analytics, machine learning initiatives, and a personalized customer experience. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">Moreover, <a href="https://www.datagaps.com/data-migration-testing-automation/"><strong><span style="color: #0000ff;">data migration testing</span></strong></a> plays a pivotal role in this ecosystem, safeguarding against potential data loss, corruption, or inconsistencies that could derail the migration project. Organizations can mitigate risks, adhere to compliance standards, and ensure a smooth migration to the new cloud by employing rigorous testing methodologies. </span></p><p><span data-contrast="none">As we delve deeper into the significance of data migration within the broader scope of business strategy, it becomes evident that mastering this domain is not just about managing data but about architecting the future of enterprises in the digital age.</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-2b82110 e-flex e-con-boxed e-con e-parent" data-id="2b82110" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-f8ad6fe elementor-widget elementor-widget-heading" data-id="f8ad6fe" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Top 6 Challenges in Automated Data Migration</h2> </div>
</div>
<div class="elementor-element elementor-element-78a0271 elementor-widget elementor-widget-text-editor" data-id="78a0271" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW88613755 BCX0">The journey of traditional data migration, </span><span class="NormalTextRun SCXW88613755 BCX0">predominantly manual</span><span class="NormalTextRun SCXW88613755 BCX0">, is beset with hurdles that can complicate and prolong the migration process, potentially jeopardizing the entire data </span><span class="NormalTextRun SCXW88613755 BCX0">m</span><span class="NormalTextRun SCXW88613755 BCX0">igra</span><span class="NormalTextRun SCXW88613755 BCX0">tion</span><span class="NormalTextRun SCXW88613755 BCX0"> strategy.</span></p> </div>
</div>
<div class="elementor-element elementor-element-4b5732b elementor-widget elementor-widget-heading" data-id="4b5732b" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">1. Data Integrity Discrepancies</h3> </div>
</div>
<div class="elementor-element elementor-element-a9a5ed1 elementor-widget elementor-widget-text-editor" data-id="a9a5ed1" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW230006839 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW230006839 BCX0">Ensuring the accuracy and consistency of data during and after migration is a common challenge. Manual migration processes are prone to errors without automated checks, leading to data discrepancies affecting business insights and decision-making. </span></span><span class="EOP SCXW230006839 BCX0" data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-5352eb1 elementor-widget elementor-widget-heading" data-id="5352eb1" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">2. Prolonged Timelines</h3> </div>
</div>
<div class="elementor-element elementor-element-ac18cf9 elementor-widget elementor-widget-text-editor" data-id="ac18cf9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW222054642 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW222054642 BCX0">Manual data migration can be exceedingly time-consuming. The process involves meticulous planning, execution, and verification stages, which can extend project timelines without automation and delay the realization of business benefits.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-cc69798 elementor-widget elementor-widget-heading" data-id="cc69798" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">3. Significant Resource Allocation</h3> </div>
</div>
<div class="elementor-element elementor-element-a157956 elementor-widget elementor-widget-text-editor" data-id="a157956" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW236928801 BCX0">Traditional data migration demands substantial human and financial resources. Skilled personnel are needed to design, implement, and </span><span class="NormalTextRun SCXW236928801 BCX0">monitor</span><span class="NormalTextRun SCXW236928801 BCX0"> the migration, diverting valuable resources from other critical business activities.</span></p> </div>
</div>
<div class="elementor-element elementor-element-58c75f0 elementor-widget elementor-widget-heading" data-id="58c75f0" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">4. Scalability Issues</h3> </div>
</div>
<div class="elementor-element elementor-element-bcc75ba elementor-widget elementor-widget-text-editor" data-id="bcc75ba" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW147607319 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW147607319 BCX0">Manual migration processes must scale with the growing size and complexity of data environments. As businesses accumulate more data, these processes become increasingly untenable, affecting the organization’s ability to adapt and grow.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-dc60e8a elementor-widget elementor-widget-heading" data-id="dc60e8a" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">5. Compliance and Security Risks</h3> </div>
</div>
<div class="elementor-element elementor-element-17a1b2d elementor-widget elementor-widget-text-editor" data-id="17a1b2d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW138052098 BCX0">Maintaining</span><span class="NormalTextRun SCXW138052098 BCX0"> data privacy and meeting regulatory compliance requirements is challenging in manual migration scenarios. Data breaches or non-compliance risk increases without robust mechanisms to protect sensitive information.</span></p> </div>
</div>
<div class="elementor-element elementor-element-893cb90 elementor-widget elementor-widget-heading" data-id="893cb90" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">6. Integration Hurdles</h3> </div>
</div>
<div class="elementor-element elementor-element-d8be94c elementor-widget elementor-widget-text-editor" data-id="d8be94c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW266879155 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW266879155 BCX0">Integrating migrated data with existing or new systems and applications often presents significant challenges. Manual interventions may lead to inconsistencies and integration failures, disrupting business processes and data flows.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-0225f06 elementor-widget elementor-widget-text-editor" data-id="0225f06" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW114206800 BCX0">These challenges underscore the need for a more sophisticated approach to data migration that </span><span class="NormalTextRun SCXW114206800 BCX0">leverages</span><span class="NormalTextRun SCXW114206800 BCX0"> automated testing and validation to ensure data integrity, reduce migration timelines, and minimize resource expenditure. The transition towards automated data migration testing marks a pivotal step in addressing these traditional hurdles, enabling organizations to enhance their data management projects and achieve greater business agility.</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-6732336 e-flex e-con-boxed e-con e-parent" data-id="6732336" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-8f3f21f elementor-widget elementor-widget-heading" data-id="8f3f21f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">The Advent of Automated Data Migration Testing </h2> </div>
</div>
<div class="elementor-element elementor-element-f4d14fa elementor-widget elementor-widget-text-editor" data-id="f4d14fa" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW235352052 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW235352052 BCX0">Automated data migration testing, pioneered by solutions like </span></span><span class="TrackedChange SCXW235352052 BCX0"><span class="TextRun Highlight SCXW235352052 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SpellingErrorV2Themed SCXW235352052 BCX0">Datagaps</span></span></span> <span class="TrackedChange SCXW235352052 BCX0"><span class="TextRun Highlight SCXW235352052 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SpellingErrorV2Themed SCXW235352052 BCX0">DataOps</span></span></span><span class="TrackedChange SCXW235352052 BCX0"><span class="TextRun Highlight SCXW235352052 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW235352052 BCX0"> Suite</span></span></span><span class="TextRun SCXW235352052 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW235352052 BCX0">, transforms the data migration process from a labor-intensive, error-prone task into a streamlined, precision-driven operation.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-250d718 elementor-widget elementor-widget-text-editor" data-id="250d718" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><b><span data-contrast="none">– Speed:</span></b><span data-contrast="none"> Automation significantly reduces the time required for data migration processes. What took months to accomplish manually can now be executed in an instant. This speed is not just about the rapid movement of data but also about the quicker realization of business value from migration projects. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Accuracy:</span></b><span data-contrast="none"> One of the most critical advantages of automation is its impact on data integrity. Automated testing tools perform meticulous, repeatable checks that ensure data is migrated accurately without the discrepancies or errors common in manual processes. This level of accuracy is vital for businesses that rely on data for strategic decision-making and operational efficiency. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Efficiency: </span></b><span data-contrast="none">Automation streamlines the entire migration process, from initial data assessment to final validation. By automating repetitive tasks, businesses can allocate their resources more effectively, focusing on strategic activities rather than the minutiae of data migration. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Scalability:</span></b><span data-contrast="none"> Automated data migration testing tools are designed to handle data of any scale, adapting to the needs of growing businesses. Whether migrating to cloud platforms like Snowflake or Azure Synapse or integrating new data sources, automation tools scale effortlessly with your data landscape. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">–</span><b><span data-contrast="none"> Risk Mitigation:</span></b><span data-contrast="none"> With automation, the risks associated with data migration—such as data loss, corruption, or breaches—are significantly lowered. Automated tools have built-in security features and compliance checks that safeguard data throughout migration. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">–</span><b><span data-contrast="none"> Innovation:</span></b><span data-contrast="none"> Automation paves the way for innovation. By eliminating the burdens of traditional data migration, businesses can confidently explore new opportunities for data utilization, adopting advanced analytics, machine learning models, and AI-driven insights. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none"><a href="https://www.datagaps.com/dataops-suite/"><strong><span style="color: #0000ff;">Datagaps DataOps Suite</span></strong></a> exemplifies this new era of automated data migration testing, offering a comprehensive suite that provides every aspect of the migration process. From initial data assessment to post-migration validation, Datagaps provides businesses with the technology to ensure their data migration projects are successful, efficient, and secure. The advent of automated data migration testing is not just a technological advancement; it’s a strategic imperative for enterprises seeking to thrive in the data-driven age. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e86c3a1 e-flex e-con-boxed e-con e-parent" data-id="e86c3a1" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-667bc4c elementor-widget elementor-widget-heading" data-id="667bc4c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Advantages of Automated Testing in Data Migration</h2> </div>
</div>
<div class="elementor-element elementor-element-07c339d elementor-widget elementor-widget-image" data-id="07c339d" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1200" height="1200" src="https://www.datagaps.com/wp-content/uploads/Strategic-Advantages-of-Automated-Testing-in-Data-Migration.jpg" class="attachment-full size-full wp-image-26605" alt="Advantages-of-Automated-Testing-in-Data-Migration" srcset="https://www.datagaps.com/wp-content/uploads/Strategic-Advantages-of-Automated-Testing-in-Data-Migration.jpg 1200w, https://www.datagaps.com/wp-content/uploads/Strategic-Advantages-of-Automated-Testing-in-Data-Migration-300x300.jpg 300w, https://www.datagaps.com/wp-content/uploads/Strategic-Advantages-of-Automated-Testing-in-Data-Migration-1024x1024.jpg 1024w, https://www.datagaps.com/wp-content/uploads/Strategic-Advantages-of-Automated-Testing-in-Data-Migration-150x150.jpg 150w, https://www.datagaps.com/wp-content/uploads/Strategic-Advantages-of-Automated-Testing-in-Data-Migration-768x768.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px" /> </div>
</div>
<div class="elementor-element elementor-element-73b0a85 elementor-widget elementor-widget-text-editor" data-id="73b0a85" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW211285433 BCX0">Automated testing in data migration strategies ushers in a host of strategic benefits that resonate across an organization’s data management framework.</span><span class="NormalTextRun SCXW211285433 BCX0"> This approach fortifies data integrity and aligns closely with the overarching goals of enterprise architects and data governance specialists</span><span class="NormalTextRun SCXW211285433 BCX0">.</span></p> </div>
</div>
<div class="elementor-element elementor-element-a9c5800 elementor-widget elementor-widget-heading" data-id="a9c5800" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">1. Ensured Data Accuracy</h3> </div>
</div>
<div class="elementor-element elementor-element-f56e9f4 elementor-widget elementor-widget-text-editor" data-id="f56e9f4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW177739848 BCX0">Automated testing tools meticulously verify each piece of data moved during the migration process, ensuring the data’s accuracy is </span><span class="NormalTextRun SCXW177739848 BCX0">maintained</span><span class="NormalTextRun SCXW177739848 BCX0"> at every step. This is crucial in environments where even minor inaccuracies can lead to significant discrepancies in reporting and analysis.</span></p> </div>
</div>
<div class="elementor-element elementor-element-0df123e elementor-widget elementor-widget-heading" data-id="0df123e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">2. Consistency Across Data Sets</h3> </div>
</div>
<div class="elementor-element elementor-element-40a07ef elementor-widget elementor-widget-text-editor" data-id="40a07ef" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW79896260 BCX0">By standardizing the testing process, automation ensures that data </span><span class="NormalTextRun SCXW79896260 BCX0">remains</span><span class="NormalTextRun SCXW79896260 BCX0"> consistent across different datasets, platforms, and systems. This uniformity is essential for organizations that rely on integrated data sources to inform their business strategies.</span></p> </div>
</div>
<div class="elementor-element elementor-element-4d4add5 elementor-widget elementor-widget-heading" data-id="4d4add5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">3. Reliability of Data</h3> </div>
</div>
<div class="elementor-element elementor-element-ce24a13 elementor-widget elementor-widget-text-editor" data-id="ce24a13" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW53210885 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW53210885 BCX0">The reliability of data post-migration is significantly enhanced with automated testing. Organizations can trust that their data to make critical business decisions is current and correctly migrated, free from corruption or loss.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-5ba98aa elementor-widget elementor-widget-heading" data-id="5ba98aa" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">4. Risk Mitigation</h3> </div>
</div>
<div class="elementor-element elementor-element-c201ffe elementor-widget elementor-widget-text-editor" data-id="c201ffe" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW53548041 BCX0">Automated testing provides a proactive approach to </span><span class="NormalTextRun SCXW53548041 BCX0">identifying</span><span class="NormalTextRun SCXW53548041 BCX0"> potential issues before they become problematic, significantly reducing the risk associated with data migrations. This includes everything from data breaches to loss of critical information, thereby safeguarding the organization’s data assets. </span></p> </div>
</div>
<div class="elementor-element elementor-element-9c948c9 elementor-widget elementor-widget-heading" data-id="9c948c9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">5. Streamlined Compliance Adherence</h3> </div>
</div>
<div class="elementor-element elementor-element-3103394 elementor-widget elementor-widget-text-editor" data-id="3103394" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW264385482 BCX0">With regulations around data becoming increasingly stringent, automated testing ensures that data migration processes </span><span class="NormalTextRun SCXW264385482 BCX0">comply with</span><span class="NormalTextRun SCXW264385482 BCX0"> relevant laws and standards. This is particularly vital for industries that deal with sensitive information, such as insurance, </span><span class="NormalTextRun SCXW264385482 BCX0">healthcare</span><span class="NormalTextRun SCXW264385482 BCX0"> and finance, where non-compliance can result in hefty penalties.</span></p> </div>
</div>
<div class="elementor-element elementor-element-3afd9ae elementor-widget elementor-widget-heading" data-id="3afd9ae" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">6. Foundation for Innovation</h3> </div>
</div>
<div class="elementor-element elementor-element-a7b638d elementor-widget elementor-widget-text-editor" data-id="a7b638d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW50953627 BCX0">By freeing up resources previously dedicated to manual testing efforts, automated testing allows organizations to </span><span class="NormalTextRun SCXW50953627 BCX0">allocate</span><span class="NormalTextRun SCXW50953627 BCX0"> more time and energy towards innovation. Enterprises can explore new data-driven technologies, analytics models, and business intelligence strategies with the assurance that their underlying data is robust and reliable.</span></p> </div>
</div>
<div class="elementor-element elementor-element-c64a9f6 elementor-widget elementor-widget-heading" data-id="c64a9f6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">7. Enhanced Collaboration</h3> </div>
</div>
<div class="elementor-element elementor-element-02fae7e elementor-widget elementor-widget-text-editor" data-id="02fae7e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW134099878 BCX0">Automated testing tools often have features that </span><span class="NormalTextRun SCXW134099878 BCX0">facilitate</span><span class="NormalTextRun SCXW134099878 BCX0"> better team collaboration. This includes shared testing environments, centralized test management, and integrated reporting tools, making it easier for different departments to collaborate on data migration projects.</span></p> </div>
</div>
<div class="elementor-element elementor-element-377a223 elementor-widget elementor-widget-text-editor" data-id="377a223" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW224371622 BCX0">For enterprise architects and data governance experts, the strategic advantages of automated testing in data migration are clear. It ensures the integrity and reliability of the migrated data and sets </span><span class="NormalTextRun SCXW224371622 BCX0">a strong foundation</span><span class="NormalTextRun SCXW224371622 BCX0"> for future growth and innovation. As </span><span class="NormalTextRun ContextualSpellingAndGrammarErrorV2Themed SCXW224371622 BCX0">enterprises</span><span class="NormalTextRun SCXW224371622 BCX0"> the complexities of migration, the role of automated testing in ensuring successful data migration projects becomes increasingly vital.</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-14fddf8 e-flex e-con-boxed e-con e-parent" data-id="14fddf8" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-8d2fe76 elementor-widget elementor-widget-heading" data-id="8d2fe76" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Future Trends in Data Migration and Testing Automation</h2> </div>
</div>
<div class="elementor-element elementor-element-71aa0c0 elementor-widget elementor-widget-text-editor" data-id="71aa0c0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW164509568 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW164509568 BCX0">As we venture deeper into the digital age, the trajectory of data migration is unmistakably veering toward a more automated and intelligent ecosystem. The </span><span class="NormalTextRun SpellingErrorV2Themed SCXW164509568 BCX0">Datagaps</span> <span class="NormalTextRun SpellingErrorV2Themed SCXW164509568 BCX0">DataOps</span><span class="NormalTextRun SCXW164509568 BCX0"> Suite stands at the vanguard of this transformation, heralding a new era in which automated testing becomes not just a facilitator but a catalyst for innovative data management strategies.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-07a781e elementor-widget elementor-widget-heading" data-id="07a781e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Here are the future of data migration and testing automation: </h4> </div>
</div>
<div class="elementor-element elementor-element-975cb61 elementor-widget elementor-widget-text-editor" data-id="975cb61" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><b><span data-contrast="none">– Integration of AI: </span></b><span data-contrast="none">Integrating artificial intelligence into data migration tools is set to redefine the efficiency and accuracy of automated testing. These technologies can predict potential issues, recommend optimizations, and even automate the resolution process, making data migrations faster and more reliable. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Enhanced Data Observability: </span></b><span data-contrast="none">Future developments will emphasize increased data observability, allowing businesses to view their data’s health and performance in real-time comprehensively. This will facilitate quicker adjustments and improvements, ensuring data integrity throughout the migration process. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Cloud-Native Solutions:</span></b><span data-contrast="none"> As organizations migrate their data to the cloud, automated cloud-native testing tools will become indispensable. These solutions offer scalability, flexibility, and integration capabilities perfectly aligned with cloud ecosystems, such as AWS, Google Cloud, and Azure. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Greater Emphasis on Security: </span></b><span data-contrast="none">With data breaches becoming a growing concern, future data migration tools will likely incorporate more robust security features. Automated testing will be crucial in identifying vulnerabilities and ensuring data migrations comply with stringent security standards. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– No-Code Platforms:</span></b><span data-contrast="none"> The rise of no-code and low-code platforms will democratize data migration, allowing non-technical users to configure and execute automated tests. This shift will significantly reduce the barrier to entry for implementing sophisticated data migration strategies. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Customizable and Flexible Testing Frameworks:</span></b><span data-contrast="none"> Anticipate the emergence of more customizable and flexible testing frameworks that can adapt to different industries and organizations’ unique needs. This customization will enable businesses to fine-tune their automated testing processes for optimal outcomes. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><b><span data-contrast="none">– Collaboration and Integration Capabilities:</span></b><span data-contrast="none"> Future tools will likely emphasize collaboration and integration, enabling seamless interaction between data migration tools, testing automation platforms, and other IT systems. This interconnectedness will streamline data migrations and foster a more cohesive IT environment. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">The Datagaps DataOps Suite, with its innovative features and forward-thinking approach, exemplifies the future of data migration and testing automation. By staying ahead of migration trends and leveraging advanced tools, organizations can navigate the complexities of data management with greater agility and confidence, ensuring they remain competitive in an increasingly data-driven world.</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-65e7a88 e-flex e-con-boxed e-con e-parent" data-id="65e7a88" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-e408c14 elementor-widget elementor-widget-heading" data-id="e408c14" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Conclusion </h4> </div>
</div>
<div class="elementor-element elementor-element-dd9a7e9 elementor-widget elementor-widget-text-editor" data-id="dd9a7e9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="none">Data migration is constantly evolving, and automated testing is the most effective solution for managing data. Datagaps DataOps Suite is the industry leader in this area, offering a comprehensive solution that prioritizes efficiency, accuracy, and foresight in managing vast data ecosystems. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">The Datagaps DataOps Suite is a game-changer, providing a platform that significantly speeds up data migration processes, minimizes errors, and ensures data integrity that manual processes cannot achieve. Using <span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/data-ops-suite-trial-request/">Datagaps DataOps Suite</a></span> for data migration testing enables businesses to adopt a precision-driven approach, anticipate data issues, optimize workflows, and unlock the full potential of their data assets. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">Investing in Datagaps DataOps Suite can help businesses navigate data challenges confidently. Advanced automation capabilities propel organizations toward a future-ready data management strategy. Automated data migration testing is essential for enterprises aiming to lead in their sector. Join the revolution with Datagaps DataOps Suite and experience its transformative power. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">Embrace the future of data migration with Datagaps DataOps Suite. Discover how automation can transform your data management strategies for enhanced efficiency, accuracy, and strategic decision-making. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335559685":-20,"335559737":-20,"335559738":0,"335559739":0,"335559740":279}"> </span></p><p><span data-contrast="none">Click here to learn more about BI Validator and </span><strong><span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/request-a-demo/">schedule your personalized demo today</a></span></strong><span data-contrast="none"><strong><span style="color: #0000ff;">.</span></strong> Transform your data analysis with the power of data testing automation!</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-417aacf e-flex e-con-boxed e-con e-parent" data-id="417aacf" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/future-impact-of-automated-testing-on-data-migration/">The Impact of Automated Testing on Data Migration</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>DataOps Suite Feature Updates – Version 2023.4.0.0</title>
<link>https://www.datagaps.com/blog/dataops-suite-feature-updates-version-2023-4-0-0/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Sat, 06 Jan 2024 11:32:19 +0000</pubDate>
<category><![CDATA[BI Testing]]></category>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[DevOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Tableau Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=25392</guid>
<description><![CDATA[<p>Datagaps Inc. announced its latest product feature updates for the DataOps Suite Version 2023.4.0.0 – The theme of this update is Integration and Easy to apply Analysis. If you are interested in getting a demo of DataOps Suite, you can fill your details here.</p>
<p>The post <a href="https://www.datagaps.com/blog/dataops-suite-feature-updates-version-2023-4-0-0/">DataOps Suite Feature Updates – Version 2023.4.0.0</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="25392" class="elementor elementor-25392" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-fed3cc0 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="fed3cc0" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-50026b7" data-id="50026b7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d17638c elementor-widget elementor-widget-text-editor" data-id="d17638c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Datagaps DataOps Suite 4.0.0 introduces a groundbreaking feature—GIT Integration. This empowers your Data team by enabling CI/CD for DataOps testing, seamlessly integrating with GIT providers such as Azure Repos and Bitbucket.</p><div class="et_pb_with_border et_pb_row et_pb_row_0"><div class="et_pb_column et_pb_column_4_4 et_pb_column_0 et_pb_css_mix_blend_mode_passthrough et-last-child"><div class="et_pb_module et_pb_text et_pb_text_0 et_pb_text_align_justified et_pb_bg_layout_light"><div class="et_pb_text_inner"><p>If you are interested in getting a <a href="https://www.datagaps.com/request-demo/">demo</a> of DataOps Suite, <a href="https://www.datagaps.com/request-demo/">you can fill your details here.</a></p></div></div></div></div><div class="et_pb_with_border et_pb_row et_pb_row_1"><div class="et_pb_column et_pb_column_4_4 et_pb_column_1 et_pb_css_mix_blend_mode_passthrough et-last-child"><div class="et_pb_with_border et_pb_module et_pb_text et_pb_text_1 et_pb_text_align_justified et_pb_bg_layout_light"><div class="et_pb_text_inner"> </div></div></div></div> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-a96efab elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="a96efab" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4c08e4f" data-id="4c08e4f" data-element_type="column" data-e-type="column" data-settings="{"background_background":"classic"}">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8579bec elementor-widget elementor-widget-heading" data-id="8579bec" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">DataOps Suite Feature Updates – Version 2023.4.0.0
</h3> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-4d91d68 elementor-widget elementor-widget-spacer" data-id="4d91d68" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-5fb112e elementor-widget elementor-widget-heading" data-id="5fb112e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">1. Enable CI/CD for your DataOps testing with GIT integration
</h2> </div>
</div>
<div class="elementor-element elementor-element-44ad90f elementor-widget elementor-widget-image" data-id="44ad90f" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="980" height="549" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-with-GIT-providers-Azure-DevOps-and-Bitbucket.webp" class="attachment-full size-full wp-image-25396" alt="" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-with-GIT-providers-Azure-DevOps-and-Bitbucket.webp 980w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-with-GIT-providers-Azure-DevOps-and-Bitbucket-300x168.webp 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-with-GIT-providers-Azure-DevOps-and-Bitbucket-768x430.webp 768w" sizes="(max-width: 980px) 100vw, 980px" /> </div>
</div>
<div class="elementor-element elementor-element-4facd5b elementor-widget elementor-widget-text-editor" data-id="4facd5b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<center>DataOps Suite with GIT providers Azure DevOps and Bitbucket</center> </div>
</div>
<div class="elementor-element elementor-element-919dc0d elementor-widget elementor-widget-text-editor" data-id="919dc0d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataOps Suite supports various GIT providers like <strong>Azure Repos</strong> and <strong>Bitbucket</strong> to store, organize, manage, and track the dataflows of different versions, making it easier to roll back to previous versions if necessary. They provide a centralized repository to manage the dataflows in one place and make it easy for multiple team members to collaborate on maintaining the dataflows.</p><p><span class="fontSizeLarge">GIT Integration empowers the Data team to manage their ETL/ELT code, BI reports/workbooks and test cases in a single GIT repository.</span></p><h4><a href="https://help.datagaps.com/articles/#!dataops-suite/git-integration">Learn More – GIT Integration</a></h4> </div>
</div>
<div class="elementor-element elementor-element-5fbd919 elementor-widget elementor-widget-heading" data-id="5fbd919" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">2. Improved Testing Capabilities for Tableau & Power BI Reports
</h2> </div>
</div>
<div class="elementor-element elementor-element-f399ec7 elementor-widget elementor-widget-text-editor" data-id="f399ec7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
DataOps Suite 4.0.0 elevates BI Analytics with enhanced features, catering to both Tableau and Power BI users.
</div>
</div>
<div class="elementor-element elementor-element-9ae8415 elementor-widget elementor-widget-image" data-id="9ae8415" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="980" height="549" src="https://www.datagaps.com/wp-content/uploads/Improved-features-of-BI-Analytics.webp" class="attachment-full size-full wp-image-25400" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Improved-features-of-BI-Analytics.webp 980w, https://www.datagaps.com/wp-content/uploads/Improved-features-of-BI-Analytics-300x168.webp 300w, https://www.datagaps.com/wp-content/uploads/Improved-features-of-BI-Analytics-768x430.webp 768w" sizes="(max-width: 980px) 100vw, 980px" /> </div>
</div>
<div class="elementor-element elementor-element-b4984db elementor-widget elementor-widget-text-editor" data-id="b4984db" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<center>Improved features of BI Analytics</center>
</div>
</div>
<div class="elementor-element elementor-element-cb7a5f8 elementor-widget elementor-widget-text-editor" data-id="cb7a5f8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_module et_pb_text et_pb_text_9 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner">In addition to Tableau Regression, DataOps Suite now supports <strong>Tableau Upgrade</strong>, <strong>Power BI Regression</strong>, and <strong>Power BI upgrade</strong>.<p> </p><p><a href="https://help.datagaps.com/articles/#!dataops-suite/tableau-upgrade"><strong>Tableau Upgrade</strong></a> <span class="fontSizeLarge">compares the views and worksheets in the newer version with the same views and the worksheets in the older version in the same or different Tableau environments to ensure that no data or formatting has been lost.</span></p><p><span class="fontSizeLarge"><a href="https://help.datagaps.com/articles/#!dataops-suite/power-bi-regression"><strong>Power BI Regression</strong></a> performs a </span>regression testing of the Power BI dashboards by comparing the baseline version of the dashboard (PDF) with the live version. This comparison can be done for ‘text’ which highlights the changes in text even if the text is shifted by a little and ‘appearance’ which performs the pixel-to-pixel comparison and highlights text, chart, etc.,</p><p><strong><a href="https://help.datagaps.com/articles/#!dataops-suite/power-bi-regression">Power BI Upgrade</a> </strong>c<span class="fontSizeLarge">ompares the PDF report in the newer version with the same PDF report in the older version in the same or different Power BI environments to ensure that no data or formatting has been lost.</span></p><p><span class="fontSizeLarge">By automating the regression and migration testing, these components can save organizations time and effort, and help them to ensure that their dashboards and reports are trustworthy and working as expected.</span></p></div></div><div class="et_pb_module et_pb_text et_pb_text_10 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><h4><a href="https://help.datagaps.com/articles/#!dataops-suite/bi-analytics">Learn More</a></h4></div></div> </div>
</div>
<div class="elementor-element elementor-element-59198bb elementor-widget elementor-widget-heading" data-id="59198bb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">3. “Data Observability” Enhancements</h2> </div>
</div>
<div class="elementor-element elementor-element-0db40ec elementor-widget elementor-widget-text-editor" data-id="0db40ec" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataOps Suite introduces two innovative detection techniques—Fixed and Delta deviation—for identifying data anomalies.</p> </div>
</div>
<div class="elementor-element elementor-element-78115f0 elementor-widget elementor-widget-image" data-id="78115f0" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="980" height="549" src="https://www.datagaps.com/wp-content/uploads/Data-Observability-enhancements.webp" class="attachment-full size-full wp-image-25404" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Data-Observability-enhancements.webp 980w, https://www.datagaps.com/wp-content/uploads/Data-Observability-enhancements-300x168.webp 300w, https://www.datagaps.com/wp-content/uploads/Data-Observability-enhancements-768x430.webp 768w" sizes="(max-width: 980px) 100vw, 980px" /> </div>
</div>
<div class="elementor-element elementor-element-71ed35b elementor-widget elementor-widget-text-editor" data-id="71ed35b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<center>Data Anomaly techniques</center>
</div>
</div>
<div class="elementor-element elementor-element-6ded7ef elementor-widget elementor-widget-text-editor" data-id="6ded7ef" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_module et_pb_text et_pb_text_14 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner">DataOps Suite has introduced two new detection techniques, <strong>Fixed and Delta deviation,</strong> for reporting the data anomalies.<p> </p><p><strong>Fixed deviation</strong> is a simple technique where we can set a fixed threshold, and if any data point deviates from the fixed threshold, it is flagged as an anomaly.</p><p><strong>Delta deviation</strong> is a sophisticated technique <span class="fontSizeLarge">that compares each data point to the previous one. If the change in the value of the data point is more than a certain amount, then it is flagged as an anomaly.</span></p><p><span class="fontSizeLarge">Both fixed and delta deviation are valuable tools that can be used to improve the <strong>quality</strong>, <strong>accuracy</strong>, and<strong> efficiency</strong> of data ingestion projects.</span></p></div></div><div class="et_pb_module et_pb_text et_pb_text_15 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><h4><a href="https://help.datagaps.com/articles/#!dataops-suite/data-observability">Learn More</a></h4></div></div> </div>
</div>
<div class="elementor-element elementor-element-e1e0245 elementor-widget elementor-widget-heading" data-id="e1e0245" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">4. Collaboration with Microsoft Teams
</h2> </div>
</div>
<div class="elementor-element elementor-element-191dacb elementor-widget elementor-widget-text-editor" data-id="191dacb" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
DataOps Suite strengthens collaboration by seamlessly integrating with Microsoft Teams, providing real-time notifications and status updates. </div>
</div>
<div class="elementor-element elementor-element-22d5614 elementor-widget elementor-widget-image" data-id="22d5614" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="980" height="549" src="https://www.datagaps.com/wp-content/uploads/Collaboration-with-Microsoft-Teams.webp" class="attachment-full size-full wp-image-25408" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Collaboration-with-Microsoft-Teams.webp 980w, https://www.datagaps.com/wp-content/uploads/Collaboration-with-Microsoft-Teams-300x168.webp 300w, https://www.datagaps.com/wp-content/uploads/Collaboration-with-Microsoft-Teams-768x430.webp 768w" sizes="(max-width: 980px) 100vw, 980px" /> </div>
</div>
<div class="elementor-element elementor-element-10c1259 elementor-widget elementor-widget-text-editor" data-id="10c1259" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<center>Collaboration with Microsoft Teams</center>
</div>
</div>
<div class="elementor-element elementor-element-47fea03 elementor-widget elementor-widget-text-editor" data-id="47fea03" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_module et_pb_text et_pb_text_14 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><div class="et_pb_module et_pb_text et_pb_text_19 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner">DataOps Suite collaborates with <a href="https://www.microsoft.com/en-us/microsoft-teams/collaboration"><strong>Microsoft Teams</strong></a> by sending the status of Pipeline and <span class="fontSizeLarge">Data Quality rules of the Data Model execution </span>to the channels through “Webhook URL<strong>”</strong>.<p> </p><p><span class="fontSizeLarge">By integrating DataOps Suite with Microsoft Teams, team members can receive notifications about pipeline or data quality rule execution status, errors, and other important notification events directly in Microsoft Teams </span>based on the configured Teams template<span class="fontSizeLarge">. This can help to improve communication and collaboration between team members, and ensure that everyone is on the same page.</span></p></div></div><div class="et_pb_module et_pb_text et_pb_text_20 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><h4><a href="https://help.datagaps.com/articles/#!dataops-suite/collaboration-tools">Learn More</a></h4></div></div></div></div><div class="et_pb_module et_pb_text et_pb_text_15 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"> </div></div> </div>
</div>
<div class="elementor-element elementor-element-375d04c elementor-widget elementor-widget-text-editor" data-id="375d04c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_module et_pb_text et_pb_text_14 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><div class="et_pb_module et_pb_text et_pb_text_19 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><div class="et_pb_module et_pb_text et_pb_text_21 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner">For more information on other enhancements of DataOps Suite version 4.0.0, read our <a href="https://help.datagaps.com/articles/#!dataops-suite/dataops-suite-releases/a/h2_203948859">Release Notes.</a></div></div><div class="et_pb_module et_pb_text et_pb_text_22 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><a href="https://help.datagaps.com/articles/#!dataops-suite/dataops-suite-releases">Full Release Notes of DataOps Suite – Learn More</a></div></div></div></div></div></div> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-29f1738 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="29f1738" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-wide">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-64b7728" data-id="64b7728" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d000e53 elementor-widget elementor-widget-heading" data-id="d000e53" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">DataOps Suite – Free Trial</h3> </div>
</div>
<div class="elementor-element elementor-element-ee08a0b elementor-widget elementor-widget-text-editor" data-id="ee08a0b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The Datagaps’ DataOps Suite now comes with new components that add extensibility and connectivity with other applications as well as a focus on ease of creating tests by automatically creating SQL Queries and identifying anomalies based on data profile. </div>
</div>
<div class="elementor-element elementor-element-901f8d5 elementor-widget elementor-widget-heading" data-id="901f8d5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Try DataOps Suite Free for 14 days…</h4> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-51e4685" data-id="51e4685" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-45cf5f0 elementor-button-align-stretch elementor-widget elementor-widget-form" data-id="45cf5f0" data-element_type="widget" data-e-type="widget" data-settings="{"step_next_label":"Next","step_previous_label":"Previous","button_width":"100","step_type":"number_text","step_icon_shape":"circle"}" data-widget_type="form.default">
<div class="elementor-widget-container">
<form class="elementor-form" method="post" name="New Form" aria-label="New Form">
<input type="hidden" name="post_id" value="25392"/>
<input type="hidden" name="form_id" value="45cf5f0"/>
<input type="hidden" name="referer_title" value="DataOps Suite Feature Updates – Version 2023.4.0" />
<input type="hidden" name="queried_id" value="25392"/>
<div class="elementor-form-fields-wrapper elementor-labels-above">
<div class="elementor-field-type-text elementor-field-group elementor-column elementor-field-group-name elementor-col-100">
<input size="1" type="text" name="form_fields[name]" id="form-field-name" class="elementor-field elementor-size-sm elementor-field-textual" placeholder="Full name">
</div>
<div class="elementor-field-type-email elementor-field-group elementor-column elementor-field-group-email elementor-col-100 elementor-field-required">
<input size="1" type="email" name="form_fields[email]" id="form-field-email" class="elementor-field elementor-size-sm elementor-field-textual" placeholder="Email" required="required">
</div>
<div class="elementor-field-type-html elementor-field-group elementor-column elementor-field-group-field_05ac09e elementor-col-100">
<p style="color:#444444;font-size:10px;margin-bottom:0;letter-spacing:1px;font-weight:600;line-height:20px;">Please check the box below</p> </div>
<div class="elementor-field-type-checkbox elementor-field-group elementor-column elementor-field-group-field_762c09a elementor-col-100">
<div class="elementor-field-subgroup "><span class="elementor-field-option"><input type="checkbox" value="I agree Datagaps and/or its representatives to reach me for further communication about the course details and any relevant Datagaps information." id="form-field-field_762c09a-0" name="form_fields[field_762c09a]"> <label for="form-field-field_762c09a-0">I agree Datagaps and/or its representatives to reach me for further communication about the course details and any relevant Datagaps information.</label></span></div> </div>
<div class="elementor-field-group elementor-column elementor-field-type-submit elementor-col-100 e-form__buttons">
<button class="elementor-button elementor-size-sm" type="submit">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Request Free Trial</span>
</span>
</button>
</div>
</div>
</form>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/dataops-suite-feature-updates-version-2023-4-0-0/">DataOps Suite Feature Updates – Version 2023.4.0.0</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Data Profiling In Pharma Datasets Using DataOps Suite</title>
<link>https://www.datagaps.com/blog/data-profiling-in-pharma-datasets-using-dataops-suite/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 14 Feb 2023 13:26:37 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=11758</guid>
<description><![CDATA[<p>Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions.</p>
<p>The post <a href="https://www.datagaps.com/blog/data-profiling-in-pharma-datasets-using-dataops-suite/">Data Profiling In Pharma Datasets Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="11758" class="elementor elementor-11758" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-9a46834 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="9a46834" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions. Data profiling involves examining and summarizing the characteristics of a dataset in order to identify patterns, trends, and anomalies in the data. By tracking aggregations and patterns in the data, it is possible to identify potential issues or anomalies that may need to be addressed in order to improve the quality of the data. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions. Data profiling involves examining and summarizing the characteristics of a dataset in order to identify patterns, trends, and anomalies in the data. By tracking aggregations and patterns in the data, it is possible to identify potential issues or anomalies that may need to be addressed in order to improve the quality of the data.</p> </div>
</div>
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Pattern Recognition and Tracking of Keys and Strings</h5> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the pharmaceutical industry, it is common for different vendors to provide datasets that contain information on the same subjects or entities. For example, a vendor may provide a dataset containing information on clinical trial participants, while another vendor may provide a dataset containing information on patient outcomes.</p><p> </p><p>In order to accurately merge or join these datasets, it is important that the primary keys used to identify the subjects or entities are consistent. For example, if one dataset uses a 9-digit numerical key to identify participants, it is important that any other datasets that contain information on the same participants also use a 9-digit numerical key.</p><p> </p><p>If the pattern of the primary keys is not consistent, it can make it difficult or impossible to accurately link records from different datasets. This can lead to errors or incorrect analyses and can compromise the overall integrity of the data.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-0db8f78 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0db8f78" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-1d20bf4" data-id="1d20bf4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-791af0e elementor-widget elementor-widget-image" data-id="791af0e" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="212" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.png" class="attachment-large size-large wp-image-11756" alt="DataOps-Suite-Logo-1" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.png 885w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-300x99.png 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-768x254.png 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-018c404" data-id="018c404" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d3fe57d elementor-widget elementor-widget-text-editor" data-id="d3fe57d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
To ensure the consistency of primary keys in pharma datasets, it is important to regularly monitor the patterns of primary keys and identify any potential issues. The DataOps Suite’s profile tracking node?? can be used to monitor the patterns of primary keys and alert you to any inconsistencies. This can help you ensure the quality and integrity of your pharma datasets and avoid any potential issues that could arise from inconsistent primary keys. </div>
</div>
<div class="elementor-element elementor-element-8c11b6b elementor-widget elementor-widget-spacer" data-id="8c11b6b" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-4bfe18a elementor-widget elementor-widget-text-editor" data-id="4bfe18a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
As seen in the example below, originally the only pattern seen in the datasets was a 9-digit key. However, in the latest run post, an update from the client we see a new alphanumeric pattern is also seen in the system. This might indicate a data-type change and a definite notification in data governance. </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-e4db7c6 elementor-widget elementor-widget-image" data-id="e4db7c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1374" height="373" src="https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key.png" class="attachment-1536x1536 size-1536x1536 wp-image-11759" alt="data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key" srcset="https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key.png 1374w, https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key-300x81.png 300w, https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key-1024x278.png 1024w, https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key-768x208.png 768w" sizes="(max-width: 1374px) 100vw, 1374px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data profile node result showcasing a change in the patterns of a primary key</figcaption>
</figure>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-deb51bb elementor-widget elementor-widget-heading" data-id="deb51bb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Outliers in Patient Claims and Drug Sales Datasets</h5> </div>
</div>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Outliers are values in a dataset that are significantly different from the majority of the other values. In inpatient claims and drug sales datasets, outliers can occur in various aggregates, such as averages, standard deviations, minimum values, and maximum values. </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Outliers can have a significant impact on the results of any analyses or modeling efforts, as they can distort the overall patterns or trends in the data. For example, if a dataset contains an outlier value that is significantly higher or lower than the majority of the other values, it could skew the average or standard deviation, leading to incorrect or misleading results.</p><p> </p><p><a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try DataOps Suite – Free Trial</a></p> </div>
</div>
<div class="elementor-element elementor-element-324bb87 elementor-widget elementor-widget-text-editor" data-id="324bb87" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<ul><li>A few examples of how variations in min-max values and standard deviations can help identify anomalies in patient claims and drug sales datasets:<br />If the minimum value for a dataset decreases significantly over time, it could indicate an anomaly or error in the data. For example, if the minimum value for a column containing drug prices decreases significantly from one month to the next, it could indicate that the price was entered incorrectly or that the drug is being sold at a significantly discounted rate.</li><li>If the maximum value for a dataset increases significantly over time, it could also indicate an anomaly or error in the data. For example, if the maximum value for a column containing drug prices increases significantly from one month to the next, it could indicate that the price was entered incorrectly or that the drug is being sold at a significantly inflated rate.</li><li>If the standard deviation for a dataset increases significantly over time, it could also indicate an anomaly or error in the data. For example, if the standard deviation for a column containing drug prices increases significantly from one month to the next, it could indicate that the prices are becoming more variable than expected, which could be a sign of an anomaly or error.</li></ul> </div>
</div>
<div class="elementor-element elementor-element-62f0ab3 elementor-widget elementor-widget-image" data-id="62f0ab3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="877" height="454" src="https://www.datagaps.com/wp-content/uploads/data-profile-node-results.png" class="attachment-1536x1536 size-1536x1536 wp-image-11761" alt="data-profile-node-results" srcset="https://www.datagaps.com/wp-content/uploads/data-profile-node-results.png 877w, https://www.datagaps.com/wp-content/uploads/data-profile-node-results-300x155.png 300w, https://www.datagaps.com/wp-content/uploads/data-profile-node-results-768x398.png 768w" sizes="(max-width: 877px) 100vw, 877px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Profile Node Results</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Also Read: <a href="https://www.datagaps.com/blog/data-drift-using-dataops-data-profiling/">Data Drift Using DataOps Data Profiling</a></p> </div>
</div>
<div class="elementor-element elementor-element-98c53c5 elementor-widget elementor-widget-heading" data-id="98c53c5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Distributions and List of Values Deltas</h5> </div>
</div>
<div class="elementor-element elementor-element-2955a70 elementor-widget elementor-widget-text-editor" data-id="2955a70" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
For inpatient claims and drug sales datasets, it is important to monitor the distribution of values across different columns and variables. The DataOps Suite’s profile node can provide various plots and statistics that can help you understand the distribution of values in your data. </div>
</div>
<div class="elementor-element elementor-element-fc1846b elementor-widget elementor-widget-text-editor" data-id="fc1846b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
For example, if you are analyzing a dataset containing information on patient claims, you might be interested in the distribution of diagnoses across different diagnosis codes. The profile node can provide a histogram or other plot showing the distribution of diagnosis codes, which can help you identify any patterns or trends in the data. </div>
</div>
<div class="elementor-element elementor-element-a757e12 elementor-widget elementor-widget-text-editor" data-id="a757e12" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In addition to monitoring the distribution of values, it can also be useful to monitor a list of values (LOV) deltas. LOV deltas refer to the difference between the list of values used in one dataset and the list of values used in another dataset. For example, if you are comparing a dataset of patient claims from one year to a dataset of patient claims from the previous year, you might be interested in the LOV deltas between the two datasets. </div>
</div>
<div class="elementor-element elementor-element-33aa808 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="33aa808" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
The DataOps Suite’s profile node can provide statistics on LOV deltas, which can help you identify any changes in the list of values used in your datasets. This can be useful for ensuring the quality and consistency of your data, and for identifying any potential issues or discrepancies. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-e5f9257 elementor-widget elementor-widget-text-editor" data-id="e5f9257" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>As seen below 2 examples:</strong></p><p> </p><p><strong>Example A</strong> deals with showcasing a change in the number of distinct values seen in a geography key of a patient claims dataset.</p><p> </p><p><strong>Example B</strong> showcases how the distribution of sales among different “Lines of Therapy” has been drastically changed indicating either an issue in the calculation of LOT, a change in behavior of the LOT in the drug in question, or worse a bug in the ETL.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-2e1dce7 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="2e1dce7" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-b54c24b" data-id="b54c24b" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8539833 elementor-widget elementor-widget-image" data-id="8539833" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="876" height="336" src="https://www.datagaps.com/wp-content/uploads/Example-A.png" class="attachment-1536x1536 size-1536x1536 wp-image-11763" alt="Example-A" srcset="https://www.datagaps.com/wp-content/uploads/Example-A.png 876w, https://www.datagaps.com/wp-content/uploads/Example-A-300x115.png 300w, https://www.datagaps.com/wp-content/uploads/Example-A-768x295.png 768w" sizes="(max-width: 876px) 100vw, 876px" /> </div>
</div>
<div class="elementor-element elementor-element-fca6f9b elementor-widget elementor-widget-image" data-id="fca6f9b" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="884" height="240" src="https://www.datagaps.com/wp-content/uploads/Example-A-01.png" class="attachment-1536x1536 size-1536x1536 wp-image-11764" alt="Example-A-01" srcset="https://www.datagaps.com/wp-content/uploads/Example-A-01.png 884w, https://www.datagaps.com/wp-content/uploads/Example-A-01-300x81.png 300w, https://www.datagaps.com/wp-content/uploads/Example-A-01-768x209.png 768w" sizes="(max-width: 884px) 100vw, 884px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-cd01ff0" data-id="cd01ff0" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-9f41a73 elementor-widget elementor-widget-image" data-id="9f41a73" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="953" height="660" src="https://www.datagaps.com/wp-content/uploads/Example-B.png" class="attachment-1536x1536 size-1536x1536 wp-image-11762" alt="Example-B" srcset="https://www.datagaps.com/wp-content/uploads/Example-B.png 953w, https://www.datagaps.com/wp-content/uploads/Example-B-300x208.png 300w, https://www.datagaps.com/wp-content/uploads/Example-B-768x532.png 768w" sizes="(max-width: 953px) 100vw, 953px" /> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-c2e0aa5 elementor-widget elementor-widget-heading" data-id="c2e0aa5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-80ad1bd elementor-widget elementor-widget-text-editor" data-id="80ad1bd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In conclusion, data profiling is an important step in the data preparation process, and it is especially important in the pharmaceutical industry where data quality and integrity are critical. The DataOps Suite’s profile node is a powerful tool that can help you perform data profiling on your pharma datasets, and it can provide valuable insights and help you identify any potential issues or inconsistencies.</p> </div>
</div>
<div class="elementor-element elementor-element-980d9ed elementor-widget elementor-widget-text-editor" data-id="980d9ed" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Some of the key features of the profile <span class="fontColorRed">node</span> include overview statistics, column statistics, and column distribution plots, which can all be useful in understanding the contents, structure, and quality of your data. In addition, the profile node can help you identify anomalies and outliers in your data, and it can provide statistics on LOV deltas, which can be useful for ensuring the consistency of your data.</p> </div>
</div>
<div class="elementor-element elementor-element-b9d0214 elementor-widget elementor-widget-text-editor" data-id="b9d0214" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, the DataOps Suite’s profile node is a valuable tool that can help you ensure the quality and integrity of your pharma datasets and support more accurate and reliable analyses and modeling efforts.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-c60bdbc e-flex e-con-boxed e-con e-parent" data-id="c60bdbc" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-d8d5665 e-con-full e-flex e-con e-child" data-id="d8d5665" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-4cab2cb elementor-widget elementor-widget-heading" data-id="4cab2cb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Get a Free POC scheduled today!</a></h2> </div>
</div>
</div>
<div class="elementor-element elementor-element-f4cf0ce e-con-full e-flex e-con e-child" data-id="f4cf0ce" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-d547dfe elementor-align-right elementor-widget elementor-widget-button" data-id="d547dfe" data-element_type="widget" data-e-type="widget" data-widget_type="button.default">
<div class="elementor-widget-container">
<div class="elementor-button-wrapper">
<a class="elementor-button elementor-button-link elementor-size-md" href="https://www.datagaps.com/request-demo/">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Click here</span>
</span>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/data-profiling-in-pharma-datasets-using-dataops-suite/">Data Profiling In Pharma Datasets Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Generate Complex SQL Queries Using DataOps Suite Query Builder</title>
<link>https://www.datagaps.com/blog/generate-complex-sql-queries-using-dataops-suite-query-builder/</link>
<comments>https://www.datagaps.com/blog/generate-complex-sql-queries-using-dataops-suite-query-builder/#respond</comments>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 14 Feb 2023 13:22:06 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=11799</guid>
<description><![CDATA[<p>An Introduction to Query Builders Query Builder is a tool that allows users to create complex SQL queries without needing in-depth knowledge of the SQL programming language. </p>
<p>The post <a href="https://www.datagaps.com/blog/generate-complex-sql-queries-using-dataops-suite-query-builder/">Generate Complex SQL Queries Using DataOps Suite Query Builder</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="11799" class="elementor elementor-11799" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-32e50a4 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="32e50a4" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-a4d046d" data-id="a4d046d" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-3e76e5d elementor-widget elementor-widget-heading" data-id="3e76e5d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">An Introduction to Query Builders</h2> </div>
</div>
<div class="elementor-element elementor-element-d26edd2 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="d26edd2" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
Query Builder is a tool that allows users to create complex SQL queries without needing in-depth knowledge of the SQL programming language. This can be especially useful for those who are new to SQL or who need to generate complex queries on a regular basis but do not have the time or expertise to write them manually. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-cec0bdc elementor-widget elementor-widget-heading" data-id="cec0bdc" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Key Benefits of using Query Builder</h2> </div>
</div>
<div class="elementor-element elementor-element-ec22005 elementor-widget elementor-widget-text-editor" data-id="ec22005" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>One of the key benefits of using Query Builder is that it allows users to build queries visually, by dragging and dropping different components such as tables, columns, and conditions into a graphical interface. This makes it easy to see how the various components of the query fit together and to make changes or adjustments as needed.</p><p>In addition to its visual interface, Query Builder also offers a number of advanced features that can help users generate more complex queries. For example, it allows users to define and save their own custom functions, which can be used in queries to perform complex calculations or operations. It also supports features such as subqueries and union queries, which can be used to combine data from multiple tables or queries in a single result set.</p> </div>
</div>
<div class="elementor-element elementor-element-dfb97f7 elementor-widget elementor-widget-heading" data-id="dfb97f7" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Complex SQL Queries for ETL Testing - Query Builder</h2> </div>
</div>
<div class="elementor-element elementor-element-bee1ad4 elementor-widget elementor-widget-image" data-id="bee1ad4" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1600" height="900" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Query-Builder.png" class="attachment-full size-full wp-image-11800" alt="DataOps-Suite-Query-Builder" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Query-Builder.png 1600w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Query-Builder-300x169.png 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Query-Builder-1024x576.png 1024w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Query-Builder-768x432.png 768w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Query-Builder-1536x864.png 1536w" sizes="(max-width: 1600px) 100vw, 1600px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Query Builder</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-c4de556 elementor-widget elementor-widget-text-editor" data-id="c4de556" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>For Beginners as well as Experts: </strong>While a SQL Query Builder might seem like a tool built to help professionals outside of the Data warehousing and ETL space to work with records, a huge number of QA Testers and Data Engineers use Query Builders on a daily basis to increase their efficiency and speed of creating the required queries.</p> </div>
</div>
<div class="elementor-element elementor-element-f217b65 elementor-widget elementor-widget-text-editor" data-id="f217b65" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>For a Professional who has to produce and maintain a large number of queries on a daily basis</strong></p><ul><li><strong>First</strong>, a query builder can make it easier and faster to create complex queries. With a query builder, you can visually construct a query by selecting different clauses and options, rather than having to write out the entire query in text form. This can save time and reduce the risk of syntax errors.</li><li><strong>Second</strong>, a query builder can also help with query organization and management. Many query builders have features that allow you to save and reuse queries, as well as to share queries with others. This can make it easier to keep track of the queries that you have created and to collaborate with others on complex data analysis tasks. In the tool, past queries can be pulled up for reference, reuse, and specific checks.</li><li><strong>Third</strong>, a query builder can also provide useful tools and features that can help you to optimize your queries and improve their performance. The DataOps Suite also holds tools made specifically to stress test ETL pipelines and using the “Enable / Disable” functionality along with the Test Data Manager System, a user can easily optimize the query in question.</li><li style="list-style-type: none;"> </li></ul> </div>
</div>
<div class="elementor-element elementor-element-c865b9c elementor-widget elementor-widget-video" data-id="c865b9c" data-element_type="widget" data-e-type="widget" data-settings="{"video_type":"hosted","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="e-hosted-video elementor-wrapper elementor-open-inline">
<video class="elementor-video" src="https://www.datagaps.com/wp-content/uploads/Query-Enable-Disable-Function.mp4" controls="" preload="metadata" controlsList="nodownload"></video>
</div>
</div>
</div>
<div class="elementor-element elementor-element-a4345d9 elementor-widget elementor-widget-text-editor" data-id="a4345d9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataOps Suite: Query Enable/Disable Function</p> </div>
</div>
<div class="elementor-element elementor-element-6ef7ecb elementor-widget elementor-widget-text-editor" data-id="6ef7ecb" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>For a person unfamiliar with SQL</strong></p><ul><li>A query builder can be useful for anyone who needs to access and analyze data stored in a database. For example, a sales manager might use a query builder to create queries that extract data about sales performance, customer demographics, and other metrics that are relevant to their role.</li><li>A query builder can also be useful for anyone who needs to collaborate with others on data analysis tasks. For example, a marketing manager might use a query builder to create and share queries with their team, or to work with data analysts on complex analysis projects.</li><li>A query builder can also be useful for anyone who needs to create and manage large numbers of queries on a regular basis. For example, an HR manager might use a query builder to create and manage a collection of queries that are used to extract and analyze data about employee performance, retention, and other HR metrics.</li><li style="list-style-type: none;"> </li></ul> </div>
</div>
<div class="elementor-element elementor-element-4ef181a elementor-widget elementor-widget-text-editor" data-id="4ef181a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/blog/monitoring-your-etl-test-data-pipelines-in-production-dataops-suite/">Also Read: Monitoring Your Data Pipelines In Production using DataOps Suite</a></p> </div>
</div>
<div class="elementor-element elementor-element-ce3c315 elementor-widget elementor-widget-heading" data-id="ce3c315" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Getting to Complex SQL Queries for ETL Testing</h2> </div>
</div>
<div class="elementor-element elementor-element-40ef489 elementor-widget elementor-widget-text-editor" data-id="40ef489" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In this section, we will showcase the DataOps Suite’s Query Builder in action creating a complex query with over 6 tables and a multitude of filters, groupings, and aggregations. But before that, a quick recap of the basics.</p> </div>
</div>
<div class="elementor-element elementor-element-adee6d4 elementor-widget elementor-widget-heading" data-id="adee6d4" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Recap of Basics</h5> </div>
</div>
<div class="elementor-element elementor-element-a83b72b elementor-widget elementor-widget-text-editor" data-id="a83b72b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The traditional method of writing SQL queries is as follows –</p><ul><li>Identify the data you want to retrieve from the database.</li><li>Determine the tables in the database that contain the data you want to retrieve.</li><li>Determine the relationships between the tables, such as which tables are related through foreign keys.</li><li>Write the SELECT statement that specifies the columns you want to retrieve from the tables.</li><li>Use the JOIN clause to specify how the tables are related and to retrieve the data from multiple tables in a single query.</li><li>Use the WHERE clause to specify any conditions that must be met for a record to be included in the result set.</li><li>Use the GROUP BY and HAVING clauses to group records and specify conditions for the groups.</li><li>Use the ORDER BY clause to specify the order in which the records should be returned in the result set.</li><li>Functions and Aggregations can be added with specific clauses given that their grouping elements are defined as well.</li><li style="list-style-type: none;"> </li></ul> </div>
</div>
<div class="elementor-element elementor-element-702f40a elementor-widget elementor-widget-text-editor" data-id="702f40a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong><a href="https://www.datagaps.com/data-ops-suite-trial-request/">Try DataOps Suite – Free Trial</a></strong></p> </div>
</div>
<div class="elementor-element elementor-element-b7f7900 elementor-widget elementor-widget-text-editor" data-id="b7f7900" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>It’s important to note that these are just general steps and the exact process for writing a complex SQL query can vary depending on the specific requirements of the query. Additionally, the complexity of a SQL query can vary greatly, so the steps outlined above may not be applicable to all complex queries. It’s always a good idea to consult the documentation for the specific SQL dialect you’re using to make sure you’re using the correct syntax and features.</p> </div>
</div>
<div class="elementor-element elementor-element-682d4ad elementor-widget elementor-widget-heading" data-id="682d4ad" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Complex SQL Queries for ETL Testing</h2> </div>
</div>
<div class="elementor-element elementor-element-51f2529 elementor-widget elementor-widget-text-editor" data-id="51f2529" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The video at the end shows the tool working in real time to create the query. The representation here is to showcase the speed and efficiency of using this tool as this removes a lot of fluff that engineers have to write up before even getting to the important parts of the query. In these parts, getting the naming convention correct, making sure syntax is not just valid but also what is expected, and the correct set of parameters have been set up is an error-prone if not a time-consuming task. Here, Query Builder shines to ensure that these aspects are taken care of so that users only think of the exact logic in question.</p><p>The problem statement here is that a User has to pull a set of records. The tables in question are Promotion, Product, Channel, and Cost-related Datasets. The User has to apply multiple sets of filters across all the tables, join them on the correct parent-child keys, choose the expected columns, and validate the query before testing/running it.</p> </div>
</div>
<div class="elementor-element elementor-element-4f41f88 elementor-widget elementor-widget-video" data-id="4f41f88" data-element_type="widget" data-e-type="widget" data-settings="{"video_type":"hosted","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="e-hosted-video elementor-wrapper elementor-open-inline">
<video class="elementor-video" src="https://www.datagaps.com/wp-content/uploads/Complex-Query-Builder.mp4" controls="" preload="metadata" controlsList="nodownload"></video>
</div>
</div>
</div>
<div class="elementor-element elementor-element-4e95f47 elementor-widget elementor-widget-text-editor" data-id="4e95f47" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataOps Suite: Complex Query Builder</p> </div>
</div>
<div class="elementor-element elementor-element-dd82841 elementor-widget elementor-widget-heading" data-id="dd82841" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-00eead5 elementor-widget elementor-widget-text-editor" data-id="00eead5" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>While a SQL expert can build up the most complex of queries on a regular basis without any hiccups and a manager could ask the DE at hand to retrieve the required records from the database, having the tools to ensure that writing these queries is simple, fast, consistent, easy to implement and easy to maintain. This ensures that if an individual has the set of rules to be applied and access to the correct datasets, they can bring out the intended results without questioning syntax, joining keys, or aggregation columns.</p> </div>
</div>
<div class="elementor-element elementor-element-c5d7a2a elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="c5d7a2a" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/generate-complex-sql-queries-using-dataops-suite-query-builder/">Generate Complex SQL Queries Using DataOps Suite Query Builder</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
<wfw:commentRss>https://www.datagaps.com/blog/generate-complex-sql-queries-using-dataops-suite-query-builder/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
<enclosure url="https://www.datagaps.com/wp-content/uploads/Query-Enable-Disable-Function.mp4" length="0" type="video/mp4" />
<enclosure url="https://www.datagaps.com/wp-content/uploads/Complex-Query-Builder.mp4" length="0" type="video/mp4" />
</item>
<item>
<title>ETL Testing In Snowflake Using DataOps Suite</title>
<link>https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/</link>
<comments>https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/#respond</comments>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 14 Feb 2023 13:18:56 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=11775</guid>
<description><![CDATA[<p>ETL stands for Extract, Transform, and Load. It is the process by which data is extracted from one or more sources, transformed into compatible formats, and then loaded into a target Database or Data Warehouse.</p>
<p>The post <a href="https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/">ETL Testing In Snowflake Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="11775" class="elementor elementor-11775" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Introduction and Overview of ETL Testing Snowflake</h2> </div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.ibm.com/in-en/topics/etl">ETL</a> stands for Extract, Transform, and Load. It is the process by which data is extracted from one or more sources, transformed into compatible formats, and then loaded into a target Database or Data Warehouse. The sources may include Flat Files, Third-Party Applications, Databases, etc. <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> is necessary to ensure that data moving from external sources to the data warehouse is accurate at each point between the source and destination.</p> </div>
</div>
<div class="elementor-element elementor-element-73a039c elementor-widget elementor-widget-heading" data-id="73a039c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Purpose of ETL</h3> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL allows businesses to consolidate data from multiple databases and other sources into a single repository with the data that has been modified and used during the analysis of data. This unified data repository allows for simplified access to analysis and additional processing of the data. There are many advantages of using ETL tools for the migration of data. It reduces delivery time, reduces unnecessary expenses, makes the process easy to use, and also will be simple for data migrations. Data Integration, Data Warehousing, and Data Migration are the three common uses of ETL.</p> </div>
</div>
<div class="elementor-element elementor-element-dd5971f elementor-widget elementor-widget-heading" data-id="dd5971f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">ETL Testing Process in Snowflake</h3> </div>
</div>
<div class="elementor-element elementor-element-4bfe18a elementor-widget elementor-widget-text-editor" data-id="4bfe18a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The data will be migrated from one data warehouse to another cloud-based <a href="https://aws.amazon.com/data-warehouse/">data warehouse</a> using various steps present in ETL Testing. The multiple steps involved in this process are the extraction of data, the transformation of the data, and finally the loading of data to the different data sources. This process is essential for proper testing such the quality of data can be checked efficiently. The DataOps Suite tool can be used efficiently for ETL Testing. <a href="https://www.datagaps.com/request-demo/">Request Demo</a></p> </div>
</div>
<div class="elementor-element elementor-element-39a261d elementor-widget elementor-widget-text-editor" data-id="39a261d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<h2><strong>The various steps involved in ETL Testing are as follows:</strong></h2> </div>
</div>
<div class="elementor-element elementor-element-27ffd33 elementor-widget elementor-widget-heading" data-id="27ffd33" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Step 1: Extraction Of Data</h3> </div>
</div>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data Extraction is the first step that will be performed in the ETL Testing. In this procedure, the data will usually be extracted from the same data source, or it can be extracted from different source locations also. Here, for example, the data is extracted from the same source i.e. Snowflake, and Customer data is extracted. After extracting the data from the source location, then further the data can be transformed according to the client’s requirements.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-2ea2482 elementor-widget elementor-widget-image" data-id="2ea2482" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1320" height="697" src="https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table.png" class="attachment-full size-full wp-image-11776" alt="Data-Extraction-From-Customers-Table" srcset="https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table.png 1320w, https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table-1024x541.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Extraction-From-Customers-Table-768x406.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Extraction From Customers Table</figcaption>
</figure>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-deb51bb elementor-widget elementor-widget-heading" data-id="deb51bb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Step 2: Transformation Of Data</h3> </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>After the data is extracted from the same or different data source to the same or the other source, a few changes or transformations in the customers’ data are done. Generally, data transformations include changes in data types or other changes according to the client’s requirements.</p><p>The below screenshot depicts the Customer data that is being transformed.</p> </div>
</div>
<div class="elementor-element elementor-element-e4db7c6 elementor-widget elementor-widget-image" data-id="e4db7c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1321" height="695" src="https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component.png" class="attachment-full size-full wp-image-11777" alt="Data-Transformation-Using-SQL-Component" srcset="https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component.png 1321w, https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Transformation-Using-SQL-Component-768x404.png 768w" sizes="(max-width: 1321px) 100vw, 1321px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Transformation Using SQL Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-324bb87 elementor-widget elementor-widget-text-editor" data-id="324bb87" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once the data is transformed, <strong>data comparison </strong>can be performed to view the changes after transformation.</p> </div>
</div>
<div class="elementor-element elementor-element-62f0ab3 elementor-widget elementor-widget-image" data-id="62f0ab3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1320" height="700" src="https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component.png" class="attachment-full size-full wp-image-11778" alt="Comparison-Of-Data-Using-Data-Compare-Component" srcset="https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component.png 1320w, https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component-300x159.png 300w, https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component-1024x543.png 1024w, https://www.datagaps.com/wp-content/uploads/Comparison-Of-Data-Using-Data-Compare-Component-768x407.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Comparison Of Data Using Data Compare Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="fontSizeMediumPlus">Further, the quality of data can be checked by using the <strong>Data Rules Component. </strong></span><span class="fontSizeMediumPlus">Data quality checks are done to find out the issues in the quality of data. The <strong>Data</strong><strong> Profile Component</strong> can also be used to find out the data quality issues.</span></p> </div>
</div>
<div class="elementor-element elementor-element-2955a70 elementor-widget elementor-widget-text-editor" data-id="2955a70" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the below screenshot, the quality of data is checked by verifying the email address as well as the name string check by using different data rules in the data rules component.</p> </div>
</div>
<div class="elementor-element elementor-element-066456a elementor-widget elementor-widget-image" data-id="066456a" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1323" height="696" src="https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component.png" class="attachment-full size-full wp-image-11779" alt="Data-Quality-Check-Using-Data-Rules-Component" srcset="https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component.png 1323w, https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Quality-Check-Using-Data-Rules-Component-768x404.png 768w" sizes="(max-width: 1323px) 100vw, 1323px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Quality Check Using Data Rules Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-fc1846b elementor-widget elementor-widget-text-editor" data-id="fc1846b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data profiling is also done to check the quality of data.</p> </div>
</div>
<div class="elementor-element elementor-element-0163057 elementor-widget elementor-widget-image" data-id="0163057" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1320" height="695" src="https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component.png" class="attachment-full size-full wp-image-11780" alt="Profiling-Data-Using-Data-Profile-Component" srcset="https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component.png 1320w, https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Profiling-Data-Using-Data-Profile-Component-768x404.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Profiling Data Using Data Profile Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-9c4aaae elementor-widget elementor-widget-heading" data-id="9c4aaae" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Step 3: Loading Of The Data</h3> </div>
</div>
<div class="elementor-element elementor-element-a757e12 elementor-widget elementor-widget-text-editor" data-id="a757e12" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once the transformation of data is performed, further the data will be loaded from one source to a particular file location. Here the data is loaded by using the <strong>DB Sink component.</strong> This is the general testing process followed in the DataOps Suite tool. <a href="https://www.datagaps.com/request-demo/">Request Demo</a></p> </div>
</div>
<div class="elementor-element elementor-element-d042b15 elementor-widget elementor-widget-text-editor" data-id="d042b15" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The below screenshot depicts the data loaded to the desired data source after the data transformations are done.</p> </div>
</div>
<div class="elementor-element elementor-element-d232146 elementor-widget elementor-widget-image" data-id="d232146" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1320" height="695" src="https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component.png" class="attachment-full size-full wp-image-11781" alt="Data-Loading-Using-DB-Sink-Component" srcset="https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component.png 1320w, https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component-300x158.png 300w, https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component-1024x539.png 1024w, https://www.datagaps.com/wp-content/uploads/Data-Loading-Using-DB-Sink-Component-768x404.png 768w" sizes="(max-width: 1320px) 100vw, 1320px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Loading Using DB Sink Component</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-e5f9257 elementor-widget elementor-widget-text-editor" data-id="e5f9257" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once the ETL Testing process is completed, the reports generated need to be checked and evaluated as there will be some differences. In our DataOps Suite tool, BI Validator can be used to check and evaluate the reports.</p><p> </p><p><a href="https://www.datagaps.com/data-testing-concepts/data-warehouse-testing-checklist/">Read: Data Warehouse Testing Checklist</a></p> </div>
</div>
<div class="elementor-element elementor-element-e3a805c elementor-widget elementor-widget-heading" data-id="e3a805c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Conclusion</h3> </div>
</div>
<div class="elementor-element elementor-element-80ad1bd elementor-widget elementor-widget-text-editor" data-id="80ad1bd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> is an important process when data is transferred from one or multiple databases to another database, especially when a huge amount of data is used. It makes sure that the data loaded in the destination source is accurate enough. The step-by-step procedure of ETL Testing can be checked by using different components in our DataOps Suite tool. By using ETL Testing, the performance can be increased. Once the entire ETL Testing is completed in Snowflake, then finally the reports will be generated. The reports generated will be checked and validated finally by using the <a href="https://www.datagaps.com/bi-validator/">BI Validator</a>.</p> </div>
</div>
<div class="elementor-element elementor-element-287ddfd elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="287ddfd" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/">ETL Testing In Snowflake Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
<wfw:commentRss>https://www.datagaps.com/blog/etl-testing-in-snowflake-using-dataops-suite/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Data Quality in Financial Institutions – Partial Flattening of Mainframe Complex Files</title>
<link>https://www.datagaps.com/blog/data-quality-in-financial-institutions-partial-flattening-of-mainframe-complex-files/</link>
<comments>https://www.datagaps.com/blog/data-quality-in-financial-institutions-partial-flattening-of-mainframe-complex-files/#respond</comments>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 07 Feb 2023 09:17:13 +0000</pubDate>
<category><![CDATA[BI Testing]]></category>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=11864</guid>
<description><![CDATA[<p>Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions.</p>
<p>The post <a href="https://www.datagaps.com/blog/data-quality-in-financial-institutions-partial-flattening-of-mainframe-complex-files/">Data Quality in Financial Institutions – Partial Flattening of Mainframe Complex Files</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="11864" class="elementor elementor-11864" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">COBOL and Financial Institutes</h2> </div>
</div>
<div class="elementor-element elementor-element-9a46834 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="9a46834" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
COBOL (Common Business-Oriented Language) is a programming language that was developed in the 1950s and remains in widespread use today, particularly in the finance and banking sectors. Many financial institutions still rely on COBOL systems to manage their data and processes, even though newer technologies such as Java and Python have largely replaced COBOL in other industries. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>One of the reasons that COBOL systems are still in use is that they are extremely stable and reliable. These systems have been in operation for decades and have proven to be effective at handling large volumes of data and transactions. In addition, COBOL systems often include technologies such as VSAM (Virtual Storage Access Method) and copybooks, which help to manage and organize the data in these systems.</p> </div>
</div>
<div class="elementor-element elementor-element-41d38a4 elementor-widget elementor-widget-text-editor" data-id="41d38a4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>However, the hierarchical nature of COBOL and finance datasets can make it difficult to migrate this data to a modern system like Snowflake. In these systems, data is often organized in a tree-like structure with multiple levels of nested records. For example, a financial transaction record might contain multiple account details, each of which might contain multiple transaction details. This hierarchical structure can make it challenging to map the data to a more flat and normalized structure like that used by <a href="https://www.snowflake.com/en/">Snowflake</a>.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-0db8f78 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0db8f78" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-1d20bf4" data-id="1d20bf4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-791af0e elementor-widget elementor-widget-image" data-id="791af0e" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="212" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.png" class="attachment-large size-large wp-image-11756" alt="DataOps-Suite-Logo-1" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.png 885w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-300x99.png 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-768x254.png 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-018c404" data-id="018c404" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d3fe57d elementor-widget elementor-widget-text-editor" data-id="d3fe57d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>To address this challenge, organizations may consider using partial flattening when migrating their data. Partial flattening involves keeping some of the hierarchy in the data while still flattening out other parts. This can be done using DataOps Suite’s python functions, which allow for more granular control over the data conversion process. </p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-29c39a2 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="29c39a2" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
A quick note that the Suite also works with binary EBCDIC files. In this blog post, we focus on a COBOL system however the same can be applied to binary files. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-27beb72 elementor-widget elementor-widget-heading" data-id="27beb72" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Partial Flattening vs Complete Flattening</h5> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_module et_pb_text et_pb_text_5 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><p>Let’s say that an organization is migrating a financial transaction record from a COBOL system to Snowflake. The transaction record in the COBOL system might have the following structure:</p></div></div> </div>
</div>
<div class="elementor-element elementor-element-4bfe18a elementor-widget elementor-widget-text-editor" data-id="4bfe18a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>To address this challenge, organizations may consider using partial flattening when migrating their data. Partial flattening involves keeping some of the hierarchy in the data while still flattening out other parts. This can be done using DataOps Suite’s python functions, which allow for more granular control over the data conversion process. </p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-e4db7c6 elementor-widget elementor-widget-image" data-id="e4db7c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="527" height="290" src="https://www.datagaps.com/wp-content/uploads/BaseRecord.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11865" alt="BaseRecord" srcset="https://www.datagaps.com/wp-content/uploads/BaseRecord.webp 527w, https://www.datagaps.com/wp-content/uploads/BaseRecord-300x165.webp 300w" sizes="(max-width: 527px) 100vw, 527px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-f56c5c6" data-id="f56c5c6" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-dcdabc0 elementor-widget elementor-widget-image" data-id="dcdabc0" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="525" height="258" src="https://www.datagaps.com/wp-content/uploads/Partial-Flattening.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11866" alt="Partial-Flattening" srcset="https://www.datagaps.com/wp-content/uploads/Partial-Flattening.webp 525w, https://www.datagaps.com/wp-content/uploads/Partial-Flattening-300x147.webp 300w" sizes="(max-width: 525px) 100vw, 525px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-566149f" data-id="566149f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-873be34 elementor-widget elementor-widget-image" data-id="873be34" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="513" height="432" src="https://www.datagaps.com/wp-content/uploads/Complete-Flattening.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11867" alt="Complete-Flattening" srcset="https://www.datagaps.com/wp-content/uploads/Complete-Flattening.webp 513w, https://www.datagaps.com/wp-content/uploads/Complete-Flattening-300x253.webp 300w" sizes="(max-width: 513px) 100vw, 513px" /> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Partial Flattening allows the organization to preserve some of the hierarchical structure of the original data (e.g., the relationship between an account and its transaction details), while still flattening out other parts to make it easier to work with in Snowflake.</p> </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Alternatively, the organization could use complete flattening to convert the transaction record. In this case, the entire hierarchical structure of the original data is flattened out, resulting in a more normalized and flat structure. However, this approach may make it more difficult to understand the relationships between different parts of the data, particularly if the data contains multiple levels of hierarchy.</p> </div>
</div>
<div class="elementor-element elementor-element-62f0ab3 elementor-widget elementor-widget-image" data-id="62f0ab3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="994" height="770" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Partial-Flatenning-Function.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11868" alt="DataOps-Suite-Partial-Flatenning-Function" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Partial-Flatenning-Function.webp 994w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Partial-Flatenning-Function-300x232.webp 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Partial-Flatenning-Function-768x595.webp 768w" sizes="(max-width: 994px) 100vw, 994px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. DataOps Suite Partial Flattening Function</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-40558c8 elementor-widget elementor-widget-image" data-id="40558c8" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1536" height="618" src="https://www.datagaps.com/wp-content/uploads/A-Complex-JSON-Flattened-1536x618.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11869" alt="A-Complex-JSON-Flattened" srcset="https://www.datagaps.com/wp-content/uploads/A-Complex-JSON-Flattened-1536x618.webp 1536w, https://www.datagaps.com/wp-content/uploads/A-Complex-JSON-Flattened-300x121.webp 300w, https://www.datagaps.com/wp-content/uploads/A-Complex-JSON-Flattened-1024x412.webp 1024w, https://www.datagaps.com/wp-content/uploads/A-Complex-JSON-Flattened-768x309.webp 768w, https://www.datagaps.com/wp-content/uploads/A-Complex-JSON-Flattened.webp 1600w" sizes="(max-width: 1536px) 100vw, 1536px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. A Complex JSON Flattened</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-44a1a74 elementor-widget elementor-widget-image" data-id="44a1a74" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1391" height="795" src="https://www.datagaps.com/wp-content/uploads/Completly-Flat-File-The-complex-columns-will-be-further-classified-into-new-rows.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11870" alt="Completly-Flat-File-The-complex-columns-will-be-further-classified-into-new-rows" srcset="https://www.datagaps.com/wp-content/uploads/Completly-Flat-File-The-complex-columns-will-be-further-classified-into-new-rows.webp 1391w, https://www.datagaps.com/wp-content/uploads/Completly-Flat-File-The-complex-columns-will-be-further-classified-into-new-rows-300x171.webp 300w, https://www.datagaps.com/wp-content/uploads/Completly-Flat-File-The-complex-columns-will-be-further-classified-into-new-rows-1024x585.webp 1024w, https://www.datagaps.com/wp-content/uploads/Completly-Flat-File-The-complex-columns-will-be-further-classified-into-new-rows-768x439.webp 768w" sizes="(max-width: 1391px) 100vw, 1391px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. Completely Flat File – The complex columns will be further classified into new rows</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-add7427 elementor-widget elementor-widget-image" data-id="add7427" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1390" height="792" src="https://www.datagaps.com/wp-content/uploads/Partially-Flat-File.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11871" alt="Partially-Flat-File" srcset="https://www.datagaps.com/wp-content/uploads/Partially-Flat-File.webp 1390w, https://www.datagaps.com/wp-content/uploads/Partially-Flat-File-300x171.webp 300w, https://www.datagaps.com/wp-content/uploads/Partially-Flat-File-1024x583.webp 1024w, https://www.datagaps.com/wp-content/uploads/Partially-Flat-File-768x438.webp 768w" sizes="(max-width: 1390px) 100vw, 1390px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. Partially Flat File</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>One of the key differences between partial flattening and complete flattening is the volume of data that is produced. Complete flattening involves flattening out all levels of hierarchy in the data, resulting in a more normalized and flat structure. This can result in a significantly larger volume of data, as all of the hierarchical relationships are preserved in the data. On the other hand, partial flattening involves keeping some of the hierarchy in the data while still flattening out other parts. This can result in a smaller volume of data, as some of the hierarchical relationships are removed from the data.</p> </div>
</div>
<div class="elementor-element elementor-element-0d3cd03 elementor-widget elementor-widget-heading" data-id="0d3cd03" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Data Quality Post Flattening</h5> </div>
</div>
<div class="elementor-element elementor-element-2955a70 elementor-widget elementor-widget-text-editor" data-id="2955a70" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Once a COBOL file has been converted in the Related Datasets, a multitude of traditional test cases that are difficult to implement in a the orignal complex structure. </p> </div>
</div>
<div class="elementor-element elementor-element-fc1846b elementor-widget elementor-widget-text-editor" data-id="fc1846b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>These take the forms of Duplicity check of ID within substructure, List-of-Values Domain checks, Null checks, Character checks and the various data quality checks. Many of these are present in the Datagaps’ DataOps Suite.</p> </div>
</div>
<div class="elementor-element elementor-element-a757e12 elementor-widget elementor-widget-text-editor" data-id="a757e12" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try DataOps Suite – Free Trial</a></p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-2e1dce7 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="2e1dce7" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-b54c24b" data-id="b54c24b" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8539833 elementor-widget elementor-widget-image" data-id="8539833" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="973" height="644" src="https://www.datagaps.com/wp-content/uploads/available-data-rules.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11872" alt="available-data-rules" srcset="https://www.datagaps.com/wp-content/uploads/available-data-rules.webp 973w, https://www.datagaps.com/wp-content/uploads/available-data-rules-300x199.webp 300w, https://www.datagaps.com/wp-content/uploads/available-data-rules-768x508.webp 768w" sizes="(max-width: 973px) 100vw, 973px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. List of available data rules in this use-cases</figcaption>
</figure>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-cd01ff0" data-id="cd01ff0" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-9f41a73 elementor-widget elementor-widget-image" data-id="9f41a73" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="685" height="695" src="https://www.datagaps.com/wp-content/uploads/Zero-Code-DQ-Node.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11873" alt="Zero-Code-DQ-Node" srcset="https://www.datagaps.com/wp-content/uploads/Zero-Code-DQ-Node.webp 685w, https://www.datagaps.com/wp-content/uploads/Zero-Code-DQ-Node-296x300.webp 296w" sizes="(max-width: 685px) 100vw, 685px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. How a Zero-Code DQ node looks in the Suite</figcaption>
</figure>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-804118f elementor-widget elementor-widget-text-editor" data-id="804118f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/data-testing-concepts/database-testing/">For more Data Rules Data at Rest Testing: Read this article on Data Quality</a> </p> </div>
</div>
<div class="elementor-element elementor-element-f52b9b7 elementor-widget elementor-widget-image" data-id="f52b9b7" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1386" height="228" src="https://www.datagaps.com/wp-content/uploads/Results-of-the-Data-Rules-Check.webp" class="attachment-1536x1536 size-1536x1536 wp-image-11874" alt="Results-of-the-Data-Rules-Check" srcset="https://www.datagaps.com/wp-content/uploads/Results-of-the-Data-Rules-Check.webp 1386w, https://www.datagaps.com/wp-content/uploads/Results-of-the-Data-Rules-Check-300x49.webp 300w, https://www.datagaps.com/wp-content/uploads/Results-of-the-Data-Rules-Check-1024x168.webp 1024w, https://www.datagaps.com/wp-content/uploads/Results-of-the-Data-Rules-Check-768x126.webp 768w" sizes="(max-width: 1386px) 100vw, 1386px" /> <figcaption class="widget-image-caption wp-caption-text">Fig. Results of the Data Rules Check</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-838b917 elementor-widget elementor-widget-heading" data-id="838b917" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-980d9ed elementor-widget elementor-widget-text-editor" data-id="980d9ed" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, the migration of COBOL and finance datasets to a modern system like Snowflake can be a complex and time-consuming process. By using partial flattening and the DataOps Suite’s python functions, organizations can ensure that their data is accurately and effectively migrated, while still maintaining the hierarchical structure that is so important in these systems. This helps to validate the mainframe datasets and ensure that the data is correctly migrated to the new system, which is critical for maintaining the integrity of the data and ensuring that it is properly understood by users.</p> </div>
</div>
<div class="elementor-element elementor-element-1659015 elementor-widget elementor-widget-heading" data-id="1659015" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Get a Free POC scheduled today!</a></h2> </div>
</div>
<div class="elementor-element elementor-element-d4f2a9e elementor-widget elementor-widget-heading" data-id="d4f2a9e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Request Demo</a></h2> </div>
</div>
<div class="elementor-element elementor-element-287ddfd elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="287ddfd" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/data-quality-in-financial-institutions-partial-flattening-of-mainframe-complex-files/">Data Quality in Financial Institutions – Partial Flattening of Mainframe Complex Files</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
<wfw:commentRss>https://www.datagaps.com/blog/data-quality-in-financial-institutions-partial-flattening-of-mainframe-complex-files/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>5 Critical Steps To Achieve Trustworthy Data in your Data Warehouse</title>
<link>https://www.datagaps.com/blog/5-critical-steps-to-achieve-trustworthy-data-in-your-data-warehouse/</link>
<dc:creator><![CDATA[David Small]]></dc:creator>
<pubDate>Tue, 17 Jan 2023 10:07:05 +0000</pubDate>
<category><![CDATA[BI Testing]]></category>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=9958</guid>
<description><![CDATA[<p>Data Warehouses provide a comprehensive view of data across many different sources, and proper analysis can encourage better business decisions and problem-solving. However, building and maintaining an effective data warehouse requires careful thought and consideration</p>
<p>The post <a href="https://www.datagaps.com/blog/5-critical-steps-to-achieve-trustworthy-data-in-your-data-warehouse/">5 Critical Steps To Achieve Trustworthy Data in your Data Warehouse</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="9958" class="elementor elementor-9958" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-c266c1a elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="c266c1a" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f89d2c9" data-id="f89d2c9" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0bfbaf7 elementor-widget elementor-widget-text-editor" data-id="0bfbaf7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://aws.amazon.com/data-warehouse/">Data Warehouses</a> provide a comprehensive view of data across many different sources, and proper analysis can encourage better business decisions and problem-solving. However, building and maintaining an effective data warehouse requires careful thought and consideration to ensure that it is reliable and can withstand the business demands of the organizations that rely on it. The biggest concern for data warehouse users is whether they can trust the information it produces.</p> </div>
</div>
<div class="elementor-element elementor-element-a64546f elementor-widget elementor-widget-text-editor" data-id="a64546f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>They can use the data to make informed decisions and implement successful business strategies if the data is trustworthy. But if the data quality is terrible, stakeholders risk making bad decisions that lead to unnecessary expenses, wasted effort, staff management issues, and many other negative impacts that directly and indirectly hurt the business.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-ebfc1ea elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="ebfc1ea" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-968226c" data-id="968226c" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-05ec210 elementor-widget elementor-widget-image" data-id="05ec210" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="335" src="https://www.datagaps.com/wp-content/uploads/Accurate-01.webp" class="attachment-large size-large wp-image-9960" alt="Accurate-01" srcset="https://www.datagaps.com/wp-content/uploads/Accurate-01.webp 690w, https://www.datagaps.com/wp-content/uploads/Accurate-01-300x157.webp 300w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-784d531" data-id="784d531" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d60042f elementor-widget elementor-widget-text-editor" data-id="d60042f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Knowing the rewards and risks involved with data accuracy, the development team must devote a large part of their time to building effective data warehouse testing methods. Many organizations do not follow a consistent or comprehensive approach to testing the quality of their data warehouses.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-81ab9a0 elementor-widget elementor-widget-text-editor" data-id="81ab9a0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>They are often overwhelmed by the complexity of managing multiple data sources and all the potential issues that could arise throughout the development process. By taking a systematic approach to testing a data warehouse, developers can continually address challenges as they arise and improve the quality of their data.</p> </div>
</div>
<div class="elementor-element elementor-element-3dbed41 elementor-widget elementor-widget-text-editor" data-id="3dbed41" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Also Read: <a href="/blog/top-5-must-haves-in-your-data-testing-platform/">Top 5 Must Haves In Your Data Testing Platform</a></p> </div>
</div>
<div class="elementor-element elementor-element-671c788 elementor-widget elementor-widget-text-editor" data-id="671c788" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Here are five critical tests that should be used during development to ensure high-quality data in data warehouses.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-689d3f5 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="689d3f5" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ba8a7ee" data-id="ba8a7ee" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-7da8015 elementor-widget elementor-widget-image" data-id="7da8015" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/number-one.webp" class="attachment-large size-large wp-image-9961" alt="number-one" srcset="https://www.datagaps.com/wp-content/uploads/number-one.webp 512w, https://www.datagaps.com/wp-content/uploads/number-one-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/number-one-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-a3fadac" data-id="a3fadac" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4bfc252 elementor-widget elementor-widget-text-editor" data-id="4bfc252" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Ensure that ETL processes run reliably by continuously monitoring the data flowing into the warehouse.</strong> This type of testing is essential for large projects with many moving parts and complex dependencies between the different datasets. For example, suppose you are working with a data warehouse incorporating data from multiple source systems, perhaps storing similar data but with varying transformation steps. If you analyze your dataset without monitoring the results throughout your development cycle, you may find that output values are incorrect or might be missing altogether. As a general rule, the longer you avoid monitoring the data for accuracy, the harder it becomes to trace back to the root source of the data and identify any fixes. To overcome this, tools that continuously monitor the source data during extraction, transformation, and loading (ETL) will ensure that the data is appropriately captured, allowing you to quickly identify anomalies in the results that require further investigation.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-348d635 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="348d635" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0643243" data-id="0643243" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-43a7dee elementor-widget elementor-widget-image" data-id="43a7dee" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/2-1.webp" class="attachment-large size-large wp-image-9962" alt="2" srcset="https://www.datagaps.com/wp-content/uploads/2-1.webp 512w, https://www.datagaps.com/wp-content/uploads/2-1-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/2-1-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-daaf553" data-id="daaf553" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-59d87b8 elementor-widget elementor-widget-text-editor" data-id="59d87b8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Make sure that the output of your transformations is accurate by running validation procedures after each transformation step.</strong> For example, if you modify your inbound data multiple times from the source to the target, you should validate the expected data after each transformation. By ensuring these validations, you will be more focused on the small details you might otherwise overlook. This test will allow you to identify any problems more efficiently and to ensure that they are corrected before delivering the new data to the warehouse.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-d8b0956 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="d8b0956" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-257d60d" data-id="257d60d" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-04e675f elementor-widget elementor-widget-image" data-id="04e675f" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/number-3.webp" class="attachment-large size-large wp-image-9963" alt="three" srcset="https://www.datagaps.com/wp-content/uploads/number-3.webp 512w, https://www.datagaps.com/wp-content/uploads/number-3-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/number-3-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0c56c0e" data-id="0c56c0e" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-a46fc97 elementor-widget elementor-widget-text-editor" data-id="a46fc97" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Execute cleaning and scrubbing procedures against your incorrect data.</strong> Many source systems, especially those in use for an extended period, can contain a lot of noise in the form of incomplete or inaccurate data. For example, a CRM system might have customer records with old addresses that are no longer valid, while other records might include outdated or incomplete contact names. Although ETL test cases assume that data remains unchanged from source to target, ‘dirty’ data can skew the results of your analysis and lead to unreliable results and inflated statistics. By building tests that identify the most common inaccuracies and transform them into the correct values, you will ensure that your warehouse incorporates higher-quality data into its operation.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-d9c249a elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="d9c249a" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-d3efdc7" data-id="d3efdc7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-c53b4d9 elementor-widget elementor-widget-image" data-id="c53b4d9" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/4-1.webp" class="attachment-large size-large wp-image-9964" alt="four" srcset="https://www.datagaps.com/wp-content/uploads/4-1.webp 512w, https://www.datagaps.com/wp-content/uploads/4-1-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/4-1-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-01def82" data-id="01def82" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0da4879 elementor-widget elementor-widget-text-editor" data-id="0da4879" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Review your source system processes and configurations regularly to identify potential impacts caused by database schema changes.</strong> For example, business logic changes might require source systems to track new fields in your data warehouse, such as marketing codes or business segments. If you miss the downstream updates, you might inadvertently include invalid data in your warehouse. In addition, the relationships identified between your tables often change as new data is added and removed., causing records to be overwritten or corrupted. Regular reviews will ensure your source systems capture the correct schema modifications before they affect the quality of your data.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-c91534f elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="c91534f" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-5d95cb6" data-id="5d95cb6" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cd949cb elementor-widget elementor-widget-image" data-id="cd949cb" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/number-5.webp" class="attachment-large size-large wp-image-9965" alt="number-5" srcset="https://www.datagaps.com/wp-content/uploads/number-5.webp 512w, https://www.datagaps.com/wp-content/uploads/number-5-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/number-5-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-8190662" data-id="8190662" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4e1bec0 elementor-widget elementor-widget-text-editor" data-id="4e1bec0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Verify the rules of your data warehouse regularly by adding unique Test Cases across the various source systems.</strong> While you ideally want to map accurate data from start to finish, including specific exception logic to ‘fix’ insufficient data, you should also include deletable source test data to verify the accuracy of your transformation steps and identify any errors that might occur in the data on its journey into the warehouse. For example, you might want to add a duplicate record on a field that might never show duplicate records to see how the system performs, or you might deliberately create an undefined lookup value to demonstrate how this error is handled. These results will help you confirm that the warehouse can process the output from the source systems under all conditions and also will provide an accurate representation of the enterprise data when the time comes.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-aa9c3f4 elementor-widget elementor-widget-text-editor" data-id="aa9c3f4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Although these five test recommendations may seem straightforward, designing them can be complicated and time-consuming without the benefit of a specific starting point. At the very minimum, begin with a flexible testing approach that focuses on validating the accuracy of the data delivered to the warehouse for one <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> use case. Then add more complexity to your scripts, incorporating the other suggestions. Remember that your ultimate goal is to create a solid test plan that produces reliable data at its core and will deliver reliable and meaningful results for your stakeholders. How you reach this goal does not matter as long as you use proven, practical approaches.</p> </div>
</div>
<div class="elementor-element elementor-element-efdcfce elementor-widget elementor-widget-text-editor" data-id="efdcfce" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Fortunately, there are several tools that you can use to accelerate your test development efforts. Whether you are building data warehouse tests from scratch or building out legacy scripts that have been in use for years, DataGaps and its suite of high-performance testing tools can help you identify data quality issues and build a robust testing framework that you can replicate and reuse throughout the project lifecycle. For example, with the DataOps Suite, you can streamline the ETL testing and quality assurance process while empowering your team to easily create and manage thousands of test cases on the fly.</p> </div>
</div>
<div class="elementor-element elementor-element-86f0998 elementor-widget elementor-widget-text-editor" data-id="86f0998" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.datagaps.com/">Datagaps</a> DataOps suite has a library of ready-to-use adaptors to popular data sources, such as Talend, Informatica, SAP, and Oracle that makes it possible to automate complex test data workflows without writing a single line of code, providing an intuitive interface that allows you to build, test, and deploy pipelines with relative ease. You can generate complex data test scripts and documents quickly, guaranteeing your users receive the trusted information they need to make better and more informed business decisions.</p><h3><a href="https://www.datagaps.com/request-demo/">To know more, request a demo now!</a></h3> </div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/5-critical-steps-to-achieve-trustworthy-data-in-your-data-warehouse/">5 Critical Steps To Achieve Trustworthy Data in your Data Warehouse</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Validation Of Complex and Hierarchical JSON and XML files</title>
<link>https://www.datagaps.com/blog/validation-of-complex-and-hierarchical-json-and-xml-files/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Mon, 09 Jan 2023 11:16:39 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=6932</guid>
<description><![CDATA[<p>While most google searches for “JSON Validation” led to validation tools and articles that focus on structure, a severely under-looked topic is about validating the data within these complex files. The same applies to XML files.</p>
<p>The post <a href="https://www.datagaps.com/blog/validation-of-complex-and-hierarchical-json-and-xml-files/">Validation Of Complex and Hierarchical JSON and XML files</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="6932" class="elementor elementor-6932" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-0ee8b4d elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0ee8b4d" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-3fac8a8" data-id="3fac8a8" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-c96c9b2 elementor-widget elementor-widget-heading" data-id="c96c9b2" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Complex Structural Hierarchical JSON and XML files</h4> </div>
</div>
<div class="elementor-element elementor-element-87619ff elementor-widget elementor-widget-heading" data-id="87619ff" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Why are complex structural and deeply hierarchical JSON and XML files so difficult to validate?</h4> </div>
</div>
<div class="elementor-element elementor-element-6bd83dc elementor-widget elementor-widget-text-editor" data-id="6bd83dc" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>While most google searches for “<a href="https://www.datagaps.com/blog/etl-validator-for-data-migration-testing/">JSON Validation</a>” led to validation tools and articles that focus on structure, a severely under-looked topic is about validating the data within these complex files. The same applies to XML files. Created by and for applications to communicate a multitude of variables that are interlinked via non-linear many-to-many methods, these files become increasingly complicated, especially to untangle and validate properly.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-2dd7d5e elementor-section-content-top bw-ac raw-json elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="2dd7d5e" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-e3f80e0" data-id="e3f80e0" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d6279db elementor-widget elementor-widget-text-editor" data-id="d6279db" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Fig. Raw JSON</p> </div>
</div>
<div class="elementor-element elementor-element-eb4bc3a elementor-widget elementor-widget-image" data-id="eb4bc3a" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="300" height="281" src="https://www.datagaps.com/wp-content/uploads/Raw-JSON-1-300x281.webp" class="attachment-medium size-medium wp-image-5575" alt="Raw-JSON" srcset="https://www.datagaps.com/wp-content/uploads/Raw-JSON-1-300x281.webp 300w, https://www.datagaps.com/wp-content/uploads/Raw-JSON-1.webp 635w" sizes="(max-width: 300px) 100vw, 300px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-e91c943" data-id="e91c943" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0a1e40b elementor-widget elementor-widget-text-editor" data-id="0a1e40b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A common use case seen apart from JSON/XML data validation only is the inclusion of these file types into more traditional datasets. A typical DB with a good number of related views and tables suddenly has to ingest a set of complex JSON files. While parsing and pulling individual data values out from the JSON itself is deemed highly situation specific and requires special functions, validating these data values is usually a nightmare for the developers.</p><p>As seen in image even loading a JSON or XML as a data frame into Spark results in a complex dataset that cannot be combined with traditional datasets.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-a1e58d5 elementor-widget elementor-widget-image" data-id="a1e58d5" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="107" src="https://www.datagaps.com/wp-content/uploads/Raw-Loaded-Tables-1-1024x171.webp" class="attachment-large size-large wp-image-5595" alt="Raw-Loaded-Tables-1" srcset="https://www.datagaps.com/wp-content/uploads/Raw-Loaded-Tables-1-1024x171.webp 1024w, https://www.datagaps.com/wp-content/uploads/Raw-Loaded-Tables-1-300x50.webp 300w, https://www.datagaps.com/wp-content/uploads/Raw-Loaded-Tables-1-768x128.webp 768w, https://www.datagaps.com/wp-content/uploads/Raw-Loaded-Tables-1.webp 1345w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-da24481 elementor-widget elementor-widget-text-editor" data-id="da24481" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Fig. Raw Loaded Tables</p> </div>
</div>
<div class="elementor-element elementor-element-ff520f3 elementor-widget elementor-widget-text-editor" data-id="ff520f3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>As XMLs and JSONs get more complex, the amount of structured columns consisting of nested arrays, key pairs, and lists only increases. While this is a start, we cannot validate any of these values or use the specifics unless we use complex functions to extract individual values. Here we introduce the auto-parsing capabilities in <a href="https://www.datagaps.com/dataops-suite/dataflow/"> DataOps Suite.</a></p> </div>
</div>
<div class="elementor-element elementor-element-433e81f elementor-widget elementor-widget-heading" data-id="433e81f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Auto-Parsing and Validation of JSON and XML</h4> </div>
</div>
<div class="elementor-element elementor-element-af7a786 elementor-widget elementor-widget-text-editor" data-id="af7a786" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The DataOps Suite holds a few custom libraries that easily parse JSON and XML files with a single command. These commands flatten different types of JSONs and XMLs into a set of relations datasets. By converting the complex sub-structures of these files into separate views with related primary keys, validation, application of data rules, profiling and analysis can be applied just as views.</p> </div>
</div>
<div class="elementor-element elementor-element-8957a2f elementor-widget elementor-widget-text-editor" data-id="8957a2f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the example, the two complex sub-structures are broken into separate views with a key-relation table. As the key pairs for the various bases (names) remain consistent, the functions ensure that no excessive data is created. In the example, each type of donut has a sub-set of the pre-existing list of batters and in the tables created we see only the distinct values and a combination of ids to correctly tag the list of values.</p> </div>
</div>
<div class="elementor-element elementor-element-6521112 elementor-widget elementor-widget-text-editor" data-id="6521112" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The segregated views can either be analyzed and validated separately to check the complete list of values and key-pairs combinations or these views can be combined with filters to perform node-specific validation. </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-40ec477 elementor-section-content-top bw-ac raw-json elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="40ec477" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-wide">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-839d097" data-id="839d097" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-5277e16 elementor-widget elementor-widget-image" data-id="5277e16" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="113" height="300" src="https://www.datagaps.com/wp-content/uploads/JSON-1-113x300.webp" class="attachment-medium size-medium wp-image-5619" alt="JSON" srcset="https://www.datagaps.com/wp-content/uploads/JSON-1-113x300.webp 113w, https://www.datagaps.com/wp-content/uploads/JSON-1.webp 297w" sizes="(max-width: 113px) 100vw, 113px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-2547022" data-id="2547022" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-44463cb elementor-widget elementor-widget-text-editor" data-id="44463cb" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Breaking sub-structures into Separate Views</p> </div>
</div>
<div class="elementor-element elementor-element-640a719 elementor-widget elementor-widget-image" data-id="640a719" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="300" height="203" src="https://www.datagaps.com/wp-content/uploads/Breaking-sub-structures-into-Seperate-Views-1-300x203.webp" class="attachment-medium size-medium wp-image-5620" alt="Breaking-sub-structures-into-Seperate-Views" srcset="https://www.datagaps.com/wp-content/uploads/Breaking-sub-structures-into-Seperate-Views-1-300x203.webp 300w, https://www.datagaps.com/wp-content/uploads/Breaking-sub-structures-into-Seperate-Views-1-768x519.webp 768w, https://www.datagaps.com/wp-content/uploads/Breaking-sub-structures-into-Seperate-Views-1.webp 946w" sizes="(max-width: 300px) 100vw, 300px" /> </div>
</div>
<div class="elementor-element elementor-element-7cdf2c1 elementor-widget elementor-widget-text-editor" data-id="7cdf2c1" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>One of the most important aspects that might be overlooked is the fact that the tables and sub-tables created by the function are part of the relational DB system. That is, these datasets have assigned connection keys and relations built the instant they are made. This makes it to that any <a href="https://www.datagaps.com/etl-testing-tools/etl-validator/">Query Builder system</a>, ETL Pipeline, or DB system can instantly work with related and work with them easily without the user having to keep track of the complex relations or joining keys.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-a17b4d3 elementor-section-content-top bw-ac raw-json elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="a17b4d3" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-wide">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-951e7a2" data-id="951e7a2" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4d5933d elementor-widget elementor-widget-text-editor" data-id="4d5933d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the example, the user makes a view to taking into consideration only the combinations of “Batters” and “Toppings” of an “Old Fashioned” Donut. This is later run across a set of data rules which include a domain check on the toppings, a null check, a duplicity check, and a space character check.</p><p>Performing a similar set of tests with custom tools on specific nodes will require a framework to be developed and maintained and each new set of rules, analysis, and profiling would require a lot more setup time and maintenance resources. This also excludes the reporting system that would have to be created and connected to the customized validation system.</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-c7bf012" data-id="c7bf012" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-f5fd8ac elementor-widget elementor-widget-image" data-id="f5fd8ac" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="280" height="300" src="https://www.datagaps.com/wp-content/uploads/Donut-Rules-1-280x300.webp" class="attachment-medium size-medium wp-image-5621" alt="Donut-Rules" srcset="https://www.datagaps.com/wp-content/uploads/Donut-Rules-1-280x300.webp 280w, https://www.datagaps.com/wp-content/uploads/Donut-Rules-1.webp 331w" sizes="(max-width: 280px) 100vw, 280px" /> </div>
</div>
<div class="elementor-element elementor-element-558992a elementor-widget elementor-widget-text-editor" data-id="558992a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Fig. Donut Rules</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-fca9f1f elementor-widget elementor-widget-text-editor" data-id="fca9f1f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_column et_pb_column_3_5 et_pb_column_6 et_pb_css_mix_blend_mode_passthrough">
<div class="et_pb_module et_pb_text et_pb_text_11 et_pb_text_align_left et_pb_bg_layout_light">
<div class="et_pb_text_inner">
<p><a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try DataOps Suite – Free Trial</a></p>
</div>
</div>
</div>
<p>In the Suite, an XML file is imported into the flow, converted into relational views, then validated across a set of rules and profiling systems. XML files are notorious to use outside the intended applications and therefore there are fewer validation systems that work to validate the values. The flow showcases how a typical validation and analysis flow for an XML file will look in the Suite. </p> </div>
</div>
<div class="elementor-element elementor-element-2ea1551 elementor-widget elementor-widget-image" data-id="2ea1551" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="131" src="https://www.datagaps.com/wp-content/uploads/Overall-Flow-1-1024x210.webp" class="attachment-large size-large wp-image-5622" alt="Overall-Flow" srcset="https://www.datagaps.com/wp-content/uploads/Overall-Flow-1-1024x210.webp 1024w, https://www.datagaps.com/wp-content/uploads/Overall-Flow-1-300x62.webp 300w, https://www.datagaps.com/wp-content/uploads/Overall-Flow-1-768x158.webp 768w, https://www.datagaps.com/wp-content/uploads/Overall-Flow-1.webp 1027w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-7ace678 elementor-widget elementor-widget-text-editor" data-id="7ace678" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Fig. Overall Flow</p> </div>
</div>
<div class="elementor-element elementor-element-ddeea30 elementor-widget elementor-widget-heading" data-id="ddeea30" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">APIs with Query Builder
</h4> </div>
</div>
<div class="elementor-element elementor-element-e5f57a5 elementor-widget elementor-widget-text-editor" data-id="e5f57a5" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The most common place to find JSONs or XMLs in a relational DB or warehouses is via APIs which come up with specific use cases. And with complex JSONs or XMLs come complex queries. In these specific cases, there are a few reasons why using the query builder provided by Datagaps can be really beneficial.</p><p>First, query builders typically provide a user-friendly interface that allows users to easily access and manipulate data without having to write complex SQL code. This can be especially helpful for non-technical users who may not be familiar with SQL or may not have the technical expertise to write queries themselves. Second, query builders often support a wide range of data manipulation functions, such as filtering, sorting, and aggregation, which can be used to transform and analyze data in various ways. This can be useful for exploring and understanding large datasets, and for identifying trends and patterns within the data. In addition, query builders can also be used to extract and flatten nested data from JSON or XML files. This can be useful for breaking down complex data structures and transforming the data into a more manageable format that is easier to work with.</p><p>Overall, using a query builder to work with data from JSON or XML files that have been pulled via an API can provide a more convenient and powerful way to access and manipulate data, and can help users to unlock the full potential of the data.</p> </div>
</div>
<div class="elementor-element elementor-element-d70bd0e elementor-widget elementor-widget-image" data-id="d70bd0e" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="348" src="https://www.datagaps.com/wp-content/uploads/Query-Builder-with-Relational-Datasets-in-the-JSON-1-1024x557.webp" class="attachment-large size-large wp-image-5623" alt="Query-Builder-with-Relational-Datasets-in-the-JSON" srcset="https://www.datagaps.com/wp-content/uploads/Query-Builder-with-Relational-Datasets-in-the-JSON-1-1024x557.webp 1024w, https://www.datagaps.com/wp-content/uploads/Query-Builder-with-Relational-Datasets-in-the-JSON-1-300x163.webp 300w, https://www.datagaps.com/wp-content/uploads/Query-Builder-with-Relational-Datasets-in-the-JSON-1-768x418.webp 768w, https://www.datagaps.com/wp-content/uploads/Query-Builder-with-Relational-Datasets-in-the-JSON-1.webp 1454w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-c7115b7 elementor-widget elementor-widget-text-editor" data-id="c7115b7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Fig. Query Builder with Relational Datasets in the JSON</p> </div>
</div>
<div class="elementor-element elementor-element-74bd307 elementor-widget elementor-widget-heading" data-id="74bd307" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-b83beed elementor-widget elementor-widget-text-editor" data-id="b83beed" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Extraction and Parsing of JSONs and XMLs which are under constant updates with structural and value type changes need a flexible solution that can be easily altered to handle any of the aforementioned changes, be it a structural change, a data type change, or anomaly detection. As most DBs work entirely with default style views addition of complex columns disrupts the flow of the quality check and pipelines.</p><p>The combination of the parsing and extraction functions that create relational views working in tandem with the data quality nodes provided by the Datagaps DataOps Suite ensures that even with routine changes of the complex XML and JSON files validating and analyzing the datasets as well as detecting anomalies in these sets is only a few clicks away.</p><p>Overall, the DataGaps ETL Validator is a valuable and Top ETL Testing tool for organizations that need to ensure the quality and integrity of their data as it is transferred from one system to another. By providing fast, efficient, and accurate data validation, the Validator can help organizations avoid costly errors and improve the reliability and effectiveness of their ETL processes.</p> </div>
</div>
<div class="elementor-element elementor-element-da8453d elementor-widget elementor-widget-heading" data-id="da8453d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="/request-a-demo/">Get a Free POC scheduled today!</a></h2> </div>
</div>
<div class="elementor-element elementor-element-ef46ef7 elementor-widget elementor-widget-heading" data-id="ef46ef7" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="/request-a-demo/">Request Demo</a></h2> </div>
</div>
<div class="elementor-element elementor-element-934507e elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="934507e" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/validation-of-complex-and-hierarchical-json-and-xml-files/">Validation Of Complex and Hierarchical JSON and XML files</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Validation of Salesforce Objects, Uploads and Updates</title>
<link>https://www.datagaps.com/blog/validation-of-salesforce-objects-uploads-and-updates/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Thu, 29 Dec 2022 16:34:31 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=6967</guid>
<description><![CDATA[<p>Salesforce is a cloud-based CRM platform that helps businesses manage and analyze customer interactions and data throughout the customer lifecycle. It is used to store and organize information about customers, such as their contact details, communication history, and preferences.</p>
<p>The post <a href="https://www.datagaps.com/blog/validation-of-salesforce-objects-uploads-and-updates/">Validation of Salesforce Objects, Uploads and Updates</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="6967" class="elementor elementor-6967" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-09034a1 elementor-widget elementor-widget-heading" data-id="09034a1" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Understanding Validation of Salesforce Objects, Uploads, <br> Updates and How Datagaps DataOps Can Help</h4> </div>
</div>
<div class="elementor-element elementor-element-379fe59 elementor-widget elementor-widget-heading" data-id="379fe59" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Intro to Salesforce</h5> </div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.salesforce.com/">Salesforce</a> is a cloud-based CRM platform that helps businesses manage and analyze customer interactions and data throughout the customer lifecycle. It is used to store and organize information about customers, such as their contact details, communication history, and preferences. This data is used to provide a complete view of the customer, which can help businesses understand their needs and personalize their interactions with them.</p> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Salesforce also provides tools for managing customer relationships, including customer segmentation, lead management, and customer service. These tools help businesses to identify and prioritize their most valuable customers, track and follow up on leads, and provide timely and efficient customer service.</p> </div>
</div>
<div class="elementor-element elementor-element-665b149 elementor-widget elementor-widget-text-editor" data-id="665b149" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In addition to its CRM capabilities, Salesforce also provides a range of tools and features for data management and integration, including data import and export, data modeling, and data governance. This makes it possible for businesses to manage and analyze their customer data in a centralized location and to integrate it with other systems and applications.</p> </div>
</div>
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Production vs Development – Metadata Validation</h5> </div>
</div>
<div class="elementor-element elementor-element-7e56b2c elementor-widget elementor-widget-text-editor" data-id="7e56b2c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div><p>In a Salesforce deployment, it is common for there to be differences in metadata between the development environment (also known as a sandbox or dev environment) and the production environment (also known as prod). This is because the development environment is often used for testing and experimentation, which can result in changes to the metadata that are not intended for the production environment.</p></div><p>Some examples of metadata changes that may occur in the development environment but not be intended for the production environment include:</p> </div>
</div>
<div class="elementor-element elementor-element-665dd37 elementor-widget elementor-widget-spacer" data-id="665dd37" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-0db8f78 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0db8f78" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-1d20bf4" data-id="1d20bf4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-3eccc05 elementor-view-default elementor-widget elementor-widget-icon" data-id="3eccc05" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-copy"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-018c404" data-id="018c404" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d3fe57d elementor-widget elementor-widget-text-editor" data-id="d3fe57d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Modifying the structure of objects or fields:</strong> This could involve adding or deleting fields, or changing field data types. For example, a developer may be testing a new feature that requires adding a new field to the Account object to store additional data. If this field is not needed in the production environment, it would be important to remove it before deploying the changes to prod. </div>
</div>
<div class="elementor-element elementor-element-8c11b6b elementor-widget elementor-widget-spacer" data-id="8c11b6b" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-174db67 elementor-view-default elementor-widget elementor-widget-icon" data-id="174db67" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-file-alt"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-dfe3010" data-id="dfe3010" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d95bda2 elementor-widget elementor-widget-text-editor" data-id="d95bda2" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Changing page layouts or field-level security settings:</strong> This could involve modifying the layout of a page to display new fields or rearranging existing ones, or changing the visibility of fields based on user roles or profiles. For example, a business user may be testing a new page layout for the Account object in the dev environment, but this layout may not be ready for production. </div>
</div>
<div class="elementor-element elementor-element-c435a58 elementor-widget elementor-widget-spacer" data-id="c435a58" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-b765395 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="b765395" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-036622e" data-id="036622e" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-00c1e76 elementor-view-default elementor-widget elementor-widget-icon" data-id="00c1e76" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-share-alt"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ab56cf5" data-id="ab56cf5" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-78203fa elementor-widget elementor-widget-text-editor" data-id="78203fa" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Modifying workflow rules or approval processes:</strong> This could involve adding or modifying rules that trigger actions based on certain conditions, or changing the steps or participants in an approval process. For example, a developer may be testing a new workflow rule in the dev environment that sends an email notification when an Account is created, but this rule may not be ready for production. </div>
</div>
<div class="elementor-element elementor-element-433f832 elementor-widget elementor-widget-spacer" data-id="433f832" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-9f808f4 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="9f808f4" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0e3983a" data-id="0e3983a" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-230c376 elementor-view-default elementor-widget elementor-widget-icon" data-id="230c376" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-folder-plus"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-4dbb62c" data-id="4dbb62c" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4365065 elementor-widget elementor-widget-text-editor" data-id="4365065" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Adding or modifying custom objects or custom fields:</strong> This could involve creating new objects to store custom data or adding new fields to existing objects. For example, a business user may be testing a new custom object in the dev environment to track project tasks, but this object may not be ready for production. </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-b49a196 elementor-widget elementor-widget-spacer" data-id="b49a196" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
These changes may occur in the development environment for a variety of reasons. For example, a developer may be testing a new feature or functionality and need to make changes to the metadata to support it. Or, a business user may be exploring different options for customizing the Salesforce instance and may make a series of changes as they iterate on their design. </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Here, the Metadata Validation Node of the <a href="https://www.datagaps.com/dataops-suite/dataflow/"> DataOps Suite.</a> can be used to ensure the metadata of the Dev and Prod Objects are in sync</p> </div>
</div>
<div class="elementor-element elementor-element-3080ee4 elementor-pagination-position-inside elementor-widget elementor-widget-image-carousel" data-id="3080ee4" data-element_type="widget" data-e-type="widget" data-settings="{"slides_to_show":"1","lazyload":"yes","autoplay":"no","navigation":"dots","infinite":"yes","effect":"slide","speed":500}" data-widget_type="image-carousel.default">
<div class="elementor-widget-container">
<div class="elementor-image-carousel-wrapper swiper" role="region" aria-roledescription="carousel" aria-label="Image Carousel" dir="ltr">
<div class="elementor-image-carousel swiper-wrapper" aria-live="polite">
<div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="1 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="2 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-02-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-02" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="3 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-03-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-03" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="4 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-04-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-04" /><div class="swiper-lazy-preloader"></div></figure></div> </div>
<div class="swiper-pagination"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-819725f elementor-widget elementor-widget-text-editor" data-id="819725f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A Metadata Validation Node against Dev and Prod Salesforce Schemas</p> </div>
</div>
<div class="elementor-element elementor-element-98c53c5 elementor-widget elementor-widget-heading" data-id="98c53c5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Upload Validation</h5> </div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
There are several issues that can arise while uploading data to Salesforce, and the specific issues you may encounter can depend on the type of data you are uploading, the source of the data, and the type of Salesforce object you are uploading to. Here are a few examples of issues that can arise while uploading data to Salesforce: </div>
</div>
<div class="elementor-element elementor-element-a94c7b8 elementor-tabs-view-horizontal elementor-widget elementor-widget-tabs" data-id="a94c7b8" data-element_type="widget" data-e-type="widget" data-widget_type="tabs.default">
<div class="elementor-widget-container">
<div class="elementor-tabs">
<div class="elementor-tabs-wrapper" role="tablist" >
<div id="elementor-tab-title-1771" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="true" data-tab="1" role="tab" tabindex="0" aria-controls="elementor-tab-content-1771" aria-expanded="false">Data Formatting</div>
<div id="elementor-tab-title-1772" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="false" data-tab="2" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1772" aria-expanded="false">Data Integrity Issues</div>
<div id="elementor-tab-title-1773" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="false" data-tab="3" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1773" aria-expanded="false">Object-Specific Issues</div>
<div id="elementor-tab-title-1774" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="false" data-tab="4" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1774" aria-expanded="false">Permission Issues</div>
</div>
<div class="elementor-tabs-content-wrapper" role="tablist" aria-orientation="vertical">
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="true" data-tab="1" role="tab" tabindex="0" aria-controls="elementor-tab-content-1771" aria-expanded="false">Data Formatting</div>
<div id="elementor-tab-content-1771" class="elementor-tab-content elementor-clearfix" data-tab="1" role="tabpanel" aria-labelledby="elementor-tab-title-1771" tabindex="0" hidden="false">If the data you are uploading is not properly formatted, it may not be accepted by Salesforce. For example, if you are uploading a CSV file and the data in the file is not properly structured, Salesforce may not be able to parse the data correctly.</div>
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="false" data-tab="2" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1772" aria-expanded="false">Data Integrity Issues</div>
<div id="elementor-tab-content-1772" class="elementor-tab-content elementor-clearfix" data-tab="2" role="tabpanel" aria-labelledby="elementor-tab-title-1772" tabindex="0" hidden="hidden">If the data you are uploading contains errors or inconsistencies, it may cause issues with the integrity of your Salesforce data. For example, if you are uploading a list of leads and some of the leads are missing required fields, the upload may fail.</div>
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="false" data-tab="3" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1773" aria-expanded="false">Object-Specific Issues</div>
<div id="elementor-tab-content-1773" class="elementor-tab-content elementor-clearfix" data-tab="3" role="tabpanel" aria-labelledby="elementor-tab-title-1773" tabindex="0" hidden="hidden">Each Salesforce object has its own set of fields and requirements, and if the data you are uploading does not meet these requirements, the upload may fail. For example, if you are uploading data to the Account object and the data does not contain a value for the required “Name” field, the upload may fail.</div>
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="false" data-tab="4" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1774" aria-expanded="false">Permission Issues</div>
<div id="elementor-tab-content-1774" class="elementor-tab-content elementor-clearfix" data-tab="4" role="tabpanel" aria-labelledby="elementor-tab-title-1774" tabindex="0" hidden="hidden">If you do not have the correct permissions in Salesforce, you may not be able to upload data to certain objects or fields. For example, if you are trying to upload data to a custom object that you do not have permission to access, the upload may fail.</div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e5f9257 elementor-widget elementor-widget-text-editor" data-id="e5f9257" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The Data Compare Node can be used to pull data directly from the Salesforce Object and compare it against the file or dataset on premise as seen below. </div>
</div>
<div class="elementor-element elementor-element-9f0ffba elementor-pagination-position-inside elementor-widget elementor-widget-image-carousel" data-id="9f0ffba" data-element_type="widget" data-e-type="widget" data-settings="{"slides_to_show":"1","lazyload":"yes","autoplay":"no","navigation":"dots","infinite":"yes","effect":"slide","speed":500}" data-widget_type="image-carousel.default">
<div class="elementor-widget-container">
<div class="elementor-image-carousel-wrapper swiper" role="region" aria-roledescription="carousel" aria-label="Image Carousel" dir="ltr">
<div class="elementor-image-carousel swiper-wrapper swiper-image-stretch" aria-live="polite">
<div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="1 of 3"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Basic-Data-Compare-checking-salesforce-against-a-dataset-01.webp" alt="Basic-Data-Compare-checking-salesforce-against-a-dataset-01" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="2 of 3"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Basic-Data-Compare-checking-salesforce-against-a-dataset-02.webp" alt="Basic-Data-Compare-checking-salesforce-against-a-dataset-02" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="3 of 3"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Basic-Data-Compare-checking-salesforce-against-a-dataset-03.webp" alt="Basic-Data-Compare-checking-salesforce-against-a-dataset-03" /><div class="swiper-lazy-preloader"></div></figure></div> </div>
<div class="swiper-pagination"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-195954c elementor-widget elementor-widget-text-editor" data-id="195954c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A Basic Data Compare checking salesforce against a dataset</p> </div>
</div>
<div class="elementor-element elementor-element-73d3e1e elementor-widget elementor-widget-heading" data-id="73d3e1e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Upsert vs Update</h5> </div>
</div>
<div class="elementor-element elementor-element-80ad1bd elementor-widget elementor-widget-text-editor" data-id="80ad1bd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
It is possible to encounter issues when uploading a large set of records that contain duplicates or need to be updated rather than inserted as new records. </div>
</div>
<div class="elementor-element elementor-element-980d9ed elementor-widget elementor-widget-text-editor" data-id="980d9ed" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
If you are using the upsert function and the records you are uploading contain duplicates based on the external ID field, the upsert function will treat these as updates rather than inserts and will update the existing records with the new data. This can be problematic if you want to insert the records as new records rather than updating the existing ones. </div>
</div>
<div class="elementor-element elementor-element-b9d0214 elementor-widget elementor-widget-text-editor" data-id="b9d0214" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In this case, you may want to use the update function instead of the upsert function. The update function allows you to specify a query to select the records you want to modify, rather than relying on the external ID field to identify matching records. This can be useful if you want to update records based on criteria other than the external ID, or if you want to insert records as new records rather than updating existing ones. </div>
</div>
<div class="elementor-element elementor-element-ab5bcb1 elementor-widget elementor-widget-text-editor" data-id="ab5bcb1" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
There are several issues that can arise while uploading data to Salesforce, and the specific issues you may encounter can depend on the type of data you are uploading, the source of the data, and the type of Salesforce object you are uploading to. Here are a few examples of issues that can arise while uploading data to Salesforce: </div>
</div>
<div class="elementor-element elementor-element-4c5ccfd elementor-widget elementor-widget-text-editor" data-id="4c5ccfd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
There are several issues that can arise while uploading data to Salesforce, and the specific issues you may encounter can depend on the type of data you are uploading, the source of the data, and the type of Salesforce object you are uploading to. Here are a few examples of issues that can arise while uploading data to Salesforce: </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-869ac3e elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="869ac3e" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-3c6189e" data-id="3c6189e" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-3fcfc10 elementor-widget elementor-widget-heading" data-id="3fcfc10" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Upsert</h5> </div>
</div>
<div class="elementor-element elementor-element-0684480 elementor-widget elementor-widget-text-editor" data-id="0684480" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The “upsert” function is used to either update existing records or create new records in an object, depending on whether a matching record already exists. When using the upsert function, you specify a field in the object that will be used as the unique identifier, called the “external ID”. If a record with a matching external ID already exists, the upsert function will update that record with the new data. If no matching record is found, the upsert function will create a new record with the provided data. </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-45cb62f" data-id="45cb62f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4e25c01 elementor-widget elementor-widget-heading" data-id="4e25c01" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Update</h5> </div>
</div>
<div class="elementor-element elementor-element-6905a2c elementor-widget elementor-widget-text-editor" data-id="6905a2c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The “update” function is used to modify existing records in an object. When using the update function, you specify the records that you want to update using a query, and then specify the new field values that you want to set for those records. The update function will only modify records that already exist in the object, and will not create new records. </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-405c1ec elementor-widget elementor-widget-spacer" data-id="405c1ec" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-4f5cecc elementor-widget elementor-widget-text-editor" data-id="4f5cecc" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Both the upsert and update functions can be used to modify a single record or multiple records at once. They can be useful for updating or creating records in bulk, or for keeping data in Salesforce synchronized with data from other sources. </div>
</div>
<div class="elementor-element elementor-element-d0a9e7b elementor-widget elementor-widget-text-editor" data-id="d0a9e7b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
To solve the complex job of figuring out which records to use with Upsert and which to use Upload, the Data Compare Node comes in handy once again. Every node in the DataOps Suite has its results, and comparisons saved as views that can be called upon internally. This allows for reference and loops of the system to ensure complex solutions can be easily defined and solved in the Suite. </div>
</div>
<div class="elementor-element elementor-element-2a0c016 elementor-widget elementor-widget-text-editor" data-id="2a0c016" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In this case, post defining the Data Compare between the Salesforce Object and the CSV file in question the node creates a set of related views such as records present only in the salesforce object, records present only in the CSV file, the records marked as the differences, and more. Our focus will the dataset which is the difference and the dataset that houses records only in the CSV file. The dataset that houses records only in the CSV file has records that are not present in the salesforce object and thus needs the upsert functions to create the new records in the salesforce object. The other dataset marks the records present in the salesforce object that need to be updated in specific value sets. Here, the update function is used with a prefixed python code to identify the exact values to be updated. </div>
</div>
<div class="elementor-element elementor-element-4bb76f6 elementor-pagination-position-inside elementor-arrows-position-inside elementor-widget elementor-widget-image-carousel" data-id="4bb76f6" data-element_type="widget" data-e-type="widget" data-settings="{"slides_to_show":"1","lazyload":"yes","autoplay":"no","navigation":"both","infinite":"yes","effect":"slide","speed":500}" data-widget_type="image-carousel.default">
<div class="elementor-widget-container">
<div class="elementor-image-carousel-wrapper swiper" role="region" aria-roledescription="carousel" aria-label="Image Carousel" dir="ltr">
<div class="elementor-image-carousel swiper-wrapper" aria-live="polite">
<div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="1 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2MCwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTAxLndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="2 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2MiwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTAyLndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="3 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2MywidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTAzLndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="4 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2NCwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA0LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="5 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2NSwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA1LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="6 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2NywidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA2LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="7 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2OCwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA3LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07" /><div class="swiper-lazy-preloader"></div></figure></a></div> </div>
<div class="elementor-swiper-button elementor-swiper-button-prev" role="button" tabindex="0">
<i aria-hidden="true" class="eicon-chevron-left"></i> </div>
<div class="elementor-swiper-button elementor-swiper-button-next" role="button" tabindex="0">
<i aria-hidden="true" class="eicon-chevron-right"></i> </div>
<div class="swiper-pagination"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-ba1c73b elementor-widget elementor-widget-text-editor" data-id="ba1c73b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The different views created by the Data Compare Node and the corresponding upsert and update nodes.</p> </div>
</div>
<div class="elementor-element elementor-element-d9aff85 elementor-widget elementor-widget-text-editor" data-id="d9aff85" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Note that distinguishing between the addition of a new external ID field and updating an existing record is a highly quintessential task especially when the objects are called by CRM and reporting tools themselves where the difference between a new record and an updated record is really critical. </div>
</div>
<div class="elementor-element elementor-element-7b3ef63 elementor-widget elementor-widget-text-editor" data-id="7b3ef63" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The below image shows all views created by the Data Compare Node and the corresponding the upsert and update nodes </div>
</div>
<div class="elementor-element elementor-element-97979c6 elementor-widget elementor-widget-image" data-id="97979c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1600" height="900" src="https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01.webp" class="attachment-full size-full wp-image-5970" alt="All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01" srcset="https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01.webp 1600w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-300x169.webp 300w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-1024x576.webp 1024w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-768x432.webp 768w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-1536x864.webp 1536w" sizes="(max-width: 1600px) 100vw, 1600px" /> </div>
</div>
<div class="elementor-element elementor-element-5281753 elementor-widget elementor-widget-spacer" data-id="5281753" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-c2e0aa5 elementor-widget elementor-widget-heading" data-id="c2e0aa5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-8eb153b elementor-widget elementor-widget-text-editor" data-id="8eb153b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In conclusion, it is important to carefully validate Salesforce objects, uploads, and updates to ensure that your data is accurate and consistent. By following best practices and using the appropriate tools and techniques, such as the DataOps Suite, you can avoid common issues such as data formatting errors, data integrity problems, and object-specific issues. Whether you are working with Veeva objects, pre-sales CRM objects, or any other type of object in Salesforce, taking the time to validate your data will help you maintain the quality and reliability of your Salesforce data. </div>
</div>
<div class="elementor-element elementor-element-32e834f elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="32e834f" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-1659015 elementor-widget elementor-widget-heading" data-id="1659015" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Get a Free POC scheduled today!</a></h2> </div>
</div>
<div class="elementor-element elementor-element-d4f2a9e elementor-widget elementor-widget-heading" data-id="d4f2a9e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Request Demo</a></h2> </div>
</div>
<div class="elementor-element elementor-element-287ddfd elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="287ddfd" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/validation-of-salesforce-objects-uploads-and-updates/">Validation of Salesforce Objects, Uploads and Updates</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
</channel>
</rss>