<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
>
<channel>
<title>Dataflow Archives - Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</title>
<atom:link href="https://www.datagaps.com/blog/category/dataflow/feed/" rel="self" type="application/rss+xml" />
<link></link>
<description></description>
<lastBuildDate>Mon, 19 Jan 2026 06:57:43 +0000</lastBuildDate>
<language>en-US</language>
<sy:updatePeriod>
hourly </sy:updatePeriod>
<sy:updateFrequency>
1 </sy:updateFrequency>
<generator>https://wordpress.org/?v=6.9.4</generator>
<image>
<url>https://www.datagaps.com/wp-content/uploads/Datagaps-India-Favicon-Lite-theme-150x150.jpg</url>
<title>Dataflow Archives - Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</title>
<link></link>
<width>32</width>
<height>32</height>
</image>
<item>
<title>Top 3 ETL Testing Tools</title>
<link>https://www.datagaps.com/blog/top-3-etl-testing-tools/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Thu, 05 Jun 2025 17:38:50 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<category><![CDATA[Snowflake]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7034</guid>
<description><![CDATA[<p>ETL Testing refers to the testing, validation, and analysis of the Extraction, Transformation, and Loading Processes that are part of ETL and ELT Pipelines. As ETL testing refers to “Data-in-Motion” Testing, the unit test architecture and principles slightly differ from “Data-at-Rest” Testing (Warehouse/DB Validation).</p>
<p>The post <a href="https://www.datagaps.com/blog/top-3-etl-testing-tools/">Top 3 ETL Testing Tools</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7034" class="elementor elementor-7034" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-95ac9c1 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="95ac9c1" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-extended">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e6bbc9f" data-id="e6bbc9f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-b8c33c8 elementor-widget elementor-widget-heading" data-id="b8c33c8" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">What Is ETL Testing Tools?</h2> </div>
</div>
<div class="elementor-element elementor-element-f29b3ae elementor-widget elementor-widget-text-editor" data-id="f29b3ae" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL Testing refers to the testing, validation, and analysis of the Extraction, Transformation, and Loading Processes that are part of ETL and ELT Pipelines. As <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> refers to “Data-in-Motion” Testing, the unit test architecture and principles slightly differ from “Data-at-Rest” Testing (Warehouse/DB Validation).</p><p>For any Top ETL Testing Tool, a typical task list of an ETL Testing Model must include:</p><ul class="custom-list"><li>Data Model Review</li><li>Source Data Testing</li><li>Post-Ingestion Validation</li><li>Post-Transform Validation</li><li>Aggregation Analysis</li><li>Data Compare between Source and Target</li><li>Data Quality and Accuracy Testing in Target</li><li>Data Integrity Examination</li><li>ETL Operational Update Validation</li><li><a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL Performance Testing</a></li></ul><p>As ETL Pipelines contain most of the transformations, relations, and aggregations that will be performed, the majority of errors occur in these sets. Even with a static Database source, as functions get updated and changed the errors can creep up even in a stable ETL pipeline.</p><p>Also read: <span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/blog/etl-validator-for-data-migration-testing/">ETL Validator for Data Migration Testing</a></span></p> </div>
</div>
<div class="elementor-element elementor-element-5a2c102 elementor-widget elementor-widget-heading" data-id="5a2c102" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Why Is ETL Testing Important?</h2> </div>
</div>
<div class="elementor-element elementor-element-41f58dd elementor-widget elementor-widget-text-editor" data-id="41f58dd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>ETL testing is important for several reasons:</p><ul><li><strong>First,</strong> it ensures the integrity and reliability of the data being used in a data warehousing or business intelligence system. By verifying the accuracy and completeness of the data, ETL testing helps to ensure that the decisions made based on that data are correct and accurate.</li><li><strong>Second,</strong> ETL testing helps to identify and resolve any issues or errors in the ETL process. This can prevent data loss and improve the overall performance of the system. For example, if an ETL test detects that certain data is missing or incorrect, the issue can be quickly addressed and corrected, which can improve the quality of the data and the reliability of the system.</li><li><strong>Third,</strong> ETL testing can help to ensure compliance with industry standards and regulations. Many industries have specific requirements for the handling and processing of data, and ETL testing can help to ensure that the data being used in a data warehousing or business intelligence system meets these requirements. This can prevent fines and penalties for non-compliance, and can also help to protect the reputation of the organization.</li></ul><p>Overall, ETL testing is a critical step in the data warehousing and business intelligence process, and it is essential for ensuring the accuracy and reliability of the data being used in these systems. Hence we should more cautious in picking the right ETL Testing Tool.</p> </div>
</div>
<div class="elementor-element elementor-element-18ad620 elementor-widget elementor-widget-heading" data-id="18ad620" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-medium">Here are some Top ETL Testing Tools available in the market.
</h2> </div>
</div>
<div class="elementor-element elementor-element-aca931e elementor-widget elementor-widget-heading" data-id="aca931e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">#1 ETL Validator</h3> </div>
</div>
<div class="elementor-element elementor-element-82c79e6 elementor-widget elementor-widget-text-editor" data-id="82c79e6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong><span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/etl-testing-tools/etl-validator/">DataGaps ETL Validator</a> </span></strong>stands on top when it comes to ETL Testing Automation. This is now part of DataOps Suite.</p> </div>
</div>
<div class="elementor-element elementor-element-2f6d8a0 elementor-widget elementor-widget-image" data-id="2f6d8a0" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img fetchpriority="high" decoding="async" width="640" height="299" src="https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-1024x479.webp" class="attachment-large size-large wp-image-5333" alt="ETL-Validator-01" srcset="https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-1024x479.webp 1024w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-300x140.webp 300w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-768x359.webp 768w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01-1536x718.webp 1536w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-01.webp 1920w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-e74ffa9 elementor-widget elementor-widget-text-editor" data-id="e74ffa9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The DataGaps ETL Validator is a tool that helps organizations ensure the quality and integrity of their data as it is transferred from one system to another through the process of Extract, Transform, and Load (ETL). The ETL Validator checks the data against a set of pre-defined rules and constraints and identifies any errors or inconsistencies that may be present. This can help organizations avoid problems such as incorrect data being loaded into their systems, or data being lost or corrupted during the ETL process.</p><p>One of the key features of the DataGaps ETL Validator is its ability to handle large amounts of data quickly and efficiently. This is important because ETL processes often involve moving large volumes of data from multiple sources, and the Validator can help organizations ensure that their data is transferred accurately and without delays.</p><p>Another important feature of the DataGaps ETL Validator is its ability to identify and highlight any errors or inconsistencies in the data. This can help organizations quickly identify and fix any issues, and ensure that their data is accurate and complete. The Validator also provides detailed reports and logs, which can be used to track the progress of the ETL process and troubleshoot any problems that may arise.</p> </div>
</div>
<div class="elementor-element elementor-element-87fdce9 elementor-widget elementor-widget-image" data-id="87fdce9" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img decoding="async" width="640" height="300" src="https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-1024x480.webp" class="attachment-large size-large wp-image-5334" alt="ETL-Validator-02" srcset="https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-1024x480.webp 1024w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-300x141.webp 300w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-768x360.webp 768w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02-1536x720.webp 1536w, https://www.datagaps.com/wp-content/uploads/ETL-Validator-02.webp 1920w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-e331430 elementor-widget elementor-widget-text-editor" data-id="e331430" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, the DataGaps ETL Validator is a valuable and Top ETL Testing tool for organizations that need to ensure the quality and integrity of their data as it is transferred from one system to another. By providing fast, efficient, and accurate data validation, the Validator can help organizations avoid costly errors and improve the reliability and effectiveness of their ETL processes.</p> </div>
</div>
<div class="elementor-element elementor-element-142a5d9 elementor-widget elementor-widget-html" data-id="142a5d9" data-element_type="widget" data-e-type="widget" data-widget_type="html.default">
<div class="elementor-widget-container">
<div class="trigger-video" data-video-url="https://www.youtube.com/watch?v=GmgCYKGZn4I" style="position: relative; cursor: pointer;">
<img decoding="async" src="https://www.datagaps.com/wp-content/uploads/Top-3-ETL-Testing-Tools-Comparison.jpg" alt="Top 3 ETL Testing Tools Comparison" style="width: 100%; height: auto;border-radius:10px">
<!-- SVG Play Icon -->
<!-- Smaller SVG Play Icon -->
<div style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;">
<svg width="60px" viewBox="0 0 68 48" xmlns="http://www.w3.org/2000/svg">
<path class="ytp-large-play-button-bg"
d="M66.52,7.74c-0.78-2.93-2.49-5.41-5.42-6.19C55.79,.13,34,0,34,0S12.21,.13,6.9,1.55
C3.97,2.33,2.27,4.81,1.48,7.74C0.06,13.05,0,24,0,24s0.06,10.95,1.48,16.26c0.78,2.93,2.49,5.41,5.42,6.19
C12.21,47.87,34,48,34,48s21.79-0.13,27.1-1.55c2.93-0.78,4.64-3.26,5.42-6.19C67.94,34.95,68,24,68,24S67.94,13.05,66.52,7.74z"
fill="#f03" />
<path d="M 45,24 27,14 27,34" fill="#fff" />
</svg>
</div>
</div>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "VideoObject",
"name": "Top 3 ETL Testing Tools Comparison",
"description": "we break down everything you need to know about ETL Testing Tools, how they work, and which tools lead the market in 2026.",
"thumbnailUrl": "https://www.datagaps.com/wp-content/uploads/Top-3-ETL-Testing-Tools-Comparison.jpg",
"uploadDate": "2025-10-31T12:00:00Z",
"duration": "PT5M59S",
"publisher": {
"@type": "Organization",
"name": "Datagaps",
"logo": {
"@type": "ImageObject",
"url": "https://www.datagaps.com/wp-content/uploads/datagaps-logo.svg"
}
},
"contentUrl": "https://www.youtube.com/watch?v=GmgCYKGZn4I",
"embedUrl": "https://www.youtube.com/embed/GmgCYKGZn4I",
"interactionStatistic": {
"@type": "InteractionCounter",
"interactionType": { "@type": "http://schema.org/WatchAction" },
"userInteractionCount": "10"
},
"regionsAllowed": ["US", "CA", "IN","GB","AU","DE","FR","IT","ES","JP","CN","RU"]
}
</script> </div>
</div>
<div class="elementor-element elementor-element-d86abb7 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="d86abb7" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-2f31c82 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="2f31c82" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-4d3d257" data-id="4d3d257" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-11c46ae elementor-widget elementor-widget-text-editor" data-id="11c46ae" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/etl-validator-trial-request/">ETL Validator – Free Trial</a></span></p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-e20f0d7" data-id="e20f0d7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-f5ebf40 elementor-widget elementor-widget-text-editor" data-id="f5ebf40" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #339966;"><a style="color: #339966;" href="https://www.datagaps.com/blog/how-to-validate-etl-testing-checklist/">ETL Testing Tool Checklist</a></span></p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-b302254" data-id="b302254" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-74398c6 elementor-widget elementor-widget-text-editor" data-id="74398c6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #339966;"><a style="color: #339966;" href="https://www.youtube.com/watch?v=j4rAuW7I7Do" data-wplink-edit="true">ETL Validator – Webinar</a></span></p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-0055398 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="0055398" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-0ac692f elementor-widget elementor-widget-heading" data-id="0ac692f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">#2 QuerySurge</h3> </div>
</div>
<div class="elementor-element elementor-element-25cddc5 elementor-widget elementor-widget-text-editor" data-id="25cddc5" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>QuerySurge is a powerful ETL Testing tool designed to help businesses and organizations quickly and efficiently test and validate their data. With its intuitive interface and robust set of features, QuerySurge makes it easy to ensure that your data is accurate, complete, and ready for use.</p><p>One of the key features of QuerySurge ETL Testing Tool is its ability to automatically generate and execute test cases. This means that you can quickly and easily test your data without having to manually write and run individual test cases. QuerySurge also allows you to specify the criteria for each test, so you can tailor your tests to fit the specific needs of your organization.</p><p>Another important feature of QuerySurge is its ability to integrate with a wide range of data sources. This means that you can use QuerySurge to test data from a variety of sources, including databases, flat files, and even web services. This flexibility allows you to easily test data from multiple sources and ensure that your data is consistent and accurate across all of your systems.</p><p>In addition to its automation and data integration capabilities, QuerySurge also offers a number of other powerful features. For example, QuerySurge allows you to define and manage your test data sets, so you can easily reuse test data and maintain a consistent testing environment. QuerySurge also provides detailed reporting capabilities, so you can easily track the progress of your tests and identify any potential issues.</p> </div>
</div>
<div class="elementor-element elementor-element-3888c62 elementor-widget elementor-widget-image" data-id="3888c62" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img decoding="async" width="640" height="328" src="https://www.datagaps.com/wp-content/uploads/QuerySurge.webp" class="attachment-large size-large wp-image-5517" alt="QuerySurge" srcset="https://www.datagaps.com/wp-content/uploads/QuerySurge.webp 745w, https://www.datagaps.com/wp-content/uploads/QuerySurge-300x154.webp 300w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-007dd63 elementor-widget elementor-widget-text-editor" data-id="007dd63" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Overall, QuerySurge is a valuable tool for anyone looking to efficiently and effectively test and validate their data. With its powerful features and intuitive interface, QuerySurge makes it easy to ensure that your data is accurate and ready for use. </div>
</div>
<div class="elementor-element elementor-element-62f3f0b elementor-widget elementor-widget-heading" data-id="62f3f0b" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">#3 iCEDQ</h3> </div>
</div>
<div class="elementor-element elementor-element-a606638 elementor-widget elementor-widget-text-editor" data-id="a606638" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>iCEDQ is a powerful data quality management tool designed to help businesses and organizations ensure the accuracy and completeness of their data. With its intuitive interface and robust set of features, iCEDQ makes it easy to identify and correct data errors, ensuring that your data is clean and ready for use.</p>
<p>One of the key features of iCEDQ is its ability to automatically identify and flag potential data errors. Using a variety of algorithms and techniques, iCEDQ can quickly and accurately detect errors in your data, such as missing values, incorrect formatting, and inconsistencies. This allows you to quickly and easily identify areas where your data may be incorrect, so you can take action to fix the errors.</p>
In addition to its error detection capabilities, iCEDQ also offers a number of other powerful features. For example, iCEDQ allows you to define and manage your data quality rules, so you can easily ensure that your data meets the specific requirements of your organization. iCEDQ also provides detailed reporting capabilities, so you can track the progress of your data quality efforts and identify areas where you may need to take action. </div>
</div>
<div class="elementor-element elementor-element-c64dee0 elementor-widget elementor-widget-image" data-id="c64dee0" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="266" src="https://www.datagaps.com/wp-content/uploads/iCEDQ.webp" class="attachment-large size-large wp-image-5520" alt="iCEDQ" srcset="https://www.datagaps.com/wp-content/uploads/iCEDQ.webp 748w, https://www.datagaps.com/wp-content/uploads/iCEDQ-300x125.webp 300w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-d041cd6 elementor-widget elementor-widget-text-editor" data-id="d041cd6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, iCEDQ is a valuable tool for anyone looking to improve the quality of their data. With its powerful features and intuitive interface, iCEDQ makes it easy to identify and correct errors in your data, ensuring that it is accurate and reliable.</p> </div>
</div>
<div class="elementor-element elementor-element-f770149 elementor-widget elementor-widget-text-editor" data-id="f770149" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="text-decoration: underline;">Disclaimer</span>: The above-mentioned list is purely an outcome of the conversations and feedback received from various industry users in the ETL/Data Warehouse testing space. Any concerns or views can be shared at <a href="mailto:contact@datagaps.com">contact@datagaps.com</a></p> </div>
</div>
<div class="elementor-element elementor-element-fc6bada elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="fc6bada" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-11a47c6 e-flex e-con-boxed e-con e-parent" data-id="11a47c6" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-0b86605 e-con-full e-flex e-con e-child" data-id="0b86605" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-bbc97a9 elementor-widget elementor-widget-heading" data-id="bbc97a9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Try ETL Validator testing tool <span style="text-decoration: underline">Free for 14</span> days for your ETL Testing Automation needs. <a href="https://www.datagaps.com/etl-validator-trial-request/">Free Trial</a></h2> </div>
</div>
</div>
<div class="elementor-element elementor-element-f932ba0 e-con-full e-flex e-con e-child" data-id="f932ba0" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-ea2db0b elementor-align-right elementor-widget elementor-widget-button" data-id="ea2db0b" data-element_type="widget" data-e-type="widget" data-widget_type="button.default">
<div class="elementor-widget-container">
<div class="elementor-button-wrapper">
<a class="elementor-button elementor-button-link elementor-size-md" href="https://www.datagaps.com/request-demo/">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Request Demo</span>
</span>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/top-3-etl-testing-tools/">Top 3 ETL Testing Tools</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Automating Tableau Reports Validation: The Easy Path to Trusted Insights </title>
<link>https://www.datagaps.com/blog/automating-tableau-reports-validation-the-easy-path-to-trusted-insights/</link>
<comments>https://www.datagaps.com/blog/automating-tableau-reports-validation-the-easy-path-to-trusted-insights/#respond</comments>
<dc:creator><![CDATA[Raj Mohan Achanta]]></dc:creator>
<pubDate>Mon, 17 Feb 2025 13:14:58 +0000</pubDate>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[Tableau Testing]]></category>
<guid isPermaLink="false">https://www.datagaps.com/?p=36020</guid>
<description><![CDATA[<p>Streamline Tableau report validation with automation to enhance accuracy, consistency, and efficiency. Detect errors early and improve data integrity. </p>
<p>The post <a href="https://www.datagaps.com/blog/automating-tableau-reports-validation-the-easy-path-to-trusted-insights/">Automating Tableau Reports Validation: The Easy Path to Trusted Insights </a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="36020" class="elementor elementor-36020" data-elementor-post-type="post">
<div class="elementor-element elementor-element-b6ca141 e-flex e-con-boxed e-con e-parent" data-id="b6ca141" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-f6b6885 elementor-widget elementor-widget-text-editor" data-id="f6b6885" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto"><span style="color: #444444;">Automating Tableau Reports Validation</span> is essential to ensure data accuracy, consistency, and reliability across multiple scenarios.</span> <span data-contrast="auto"><span style="color: #444444;"><a style="color: #444444;" href="https://en.wikipedia.org/wiki/Manual_testing">Manual validation</a> </span>can be time-consuming and prone to human error, especially when dealing with complex dashboards and large datasets. By leveraging automation, organizations can streamline the validation process, quickly detect discrepancies, and enhance overall data integrity.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">Going ahead, we’ll explore automation of Tableau reports validation and how it is done.</span><span data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-ea3768c elementor-widget elementor-widget-heading" data-id="ea3768c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Importance of Automating Tableau Reports Validation</h2> </div>
</div>
<div class="elementor-element elementor-element-a3394d0 elementor-widget elementor-widget-text-editor" data-id="a3394d0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW258116418 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW258116418 BCX0">Automating Tableau report validation provides several benefits, ensuring accuracy, efficiency, and reliability in BI reporting.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-ea1b5ca elementor-widget elementor-widget-text-editor" data-id="ea1b5ca" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div style="font-family: 'Poppins', sans-serif; color: #444444;">
<ul style="list-style-type: disc; color: #444444;">
<li>Automating the reports validation reduces the time and effort, which allows analysts to focus on insights rather than troubleshooting the errors.</li>
<li>Automation prevents data discrepancies and ensures all reports are pulling in consistent data.</li>
<li>Many organizations deal with high volumes of reports and dashboards. It is difficult to manually validate each report. Automating the reports validation becomes critical to maintain efficiency.</li>
<li>Organizations update their Tableau dashboards very frequently, sometimes daily. On automating the reports validation process, a direct comparison is made between the previous and current data to detect changes or discrepancies. This ensures metrics remain consistent after each data refresh.</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e472b52 e-flex e-con-boxed e-con e-parent" data-id="e472b52" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-dd9bc66 elementor-widget elementor-widget-heading" data-id="dd9bc66" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Enabling Automation of Tableau Report Validation with DataOps Suite BI Validator</h2> </div>
</div>
<div class="elementor-element elementor-element-8077960 elementor-widget elementor-widget-image" data-id="8077960" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1200" height="628" src="https://www.datagaps.com/wp-content/uploads/Automation-of-Tableau-Report-Validation-with-DataOps-Suite-BI-Validator.jpg" class="attachment-full size-full wp-image-36117" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Automation-of-Tableau-Report-Validation-with-DataOps-Suite-BI-Validator.jpg 1200w, https://www.datagaps.com/wp-content/uploads/Automation-of-Tableau-Report-Validation-with-DataOps-Suite-BI-Validator-300x157.jpg 300w, https://www.datagaps.com/wp-content/uploads/Automation-of-Tableau-Report-Validation-with-DataOps-Suite-BI-Validator-1024x536.jpg 1024w, https://www.datagaps.com/wp-content/uploads/Automation-of-Tableau-Report-Validation-with-DataOps-Suite-BI-Validator-768x402.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px" /> </div>
</div>
<div class="elementor-element elementor-element-89e3cf8 elementor-widget elementor-widget-text-editor" data-id="89e3cf8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW80684046 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW80684046 BCX0"><span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/bi-validator/">BI Validator</a></span> simplifies BI testing by providing a platform for automated BI report testing. It enables seamless regression, stress, and performance testing, making the process faster and more reliable.</span></span><span class="EOP SCXW80684046 BCX0" data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-835096b elementor-widget elementor-widget-heading" data-id="835096b" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Tableau Reports to Database Validation </h3> </div>
</div>
<div class="elementor-element elementor-element-ec7cd02 elementor-widget elementor-widget-text-editor" data-id="ec7cd02" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">Tableau reports to Database data comparison ensures that the records from the source data are reflected accurately in the visuals of Tableau reports.</span><span data-ccp-props="{"201341983":0,"335559740":276}"> </span></p><p><span data-contrast="auto">This validation process extracts data from Tableau report visuals and </span><span data-contrast="auto">compares it with SQL Server, Oracle, Snowflake, or other databases. <a href="https://www.datagaps.com/bi-validator/"><span style="color: #0000ff;">Datagaps DataOps Suite BI Validator</span></a> streamlines this by pulling report data, applying transformations, and verifying consistency through automated row-by-row and aggregate comparisons (e.g., counts, sums, averages).</span><span data-contrast="auto"> </span><span data-ccp-props="{"201341983":0,"335559740":276}"> </span></p><p><span data-contrast="auto">The errors detected usually identify missing, duplicate or mismatched records.</span><span data-ccp-props="{"201341983":0,"335559740":276}"> </span></p><p><span data-contrast="auto">Automation ensures these issues are caught early, reducing manual effort and improving trust in reporting.</span><span data-ccp-props="{"201341983":0,"335559740":276}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-ba20992 elementor-widget elementor-widget-image" data-id="ba20992" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="602" height="168" src="https://www.datagaps.com/wp-content/uploads/Tableau-Reports-to-Database-Validation.png" class="attachment-full size-full wp-image-36083" alt="Tableau Reports to Database Validation Process" srcset="https://www.datagaps.com/wp-content/uploads/Tableau-Reports-to-Database-Validation.png 602w, https://www.datagaps.com/wp-content/uploads/Tableau-Reports-to-Database-Validation-300x84.png 300w, https://www.datagaps.com/wp-content/uploads/Tableau-Reports-to-Database-Validation-600x168.png 600w" sizes="(max-width: 602px) 100vw, 602px" /> </div>
</div>
<div class="elementor-element elementor-element-b5b8afd elementor-widget elementor-widget-heading" data-id="b5b8afd" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Tableau Regression</h3> </div>
</div>
<div class="elementor-element elementor-element-36ed43e elementor-widget elementor-widget-text-editor" data-id="36ed43e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">In the <a href="https://www.datagaps.com/dataops-suite/"><span style="color: #0000ff;">DataOps suite</span></a>, Regression testing is done by comparing the benchmarked version of tableau report with the live version of the report through Tableau Regression component.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">The comparison can be done in terms of text where the textual differences are highlighted and in terms of appearance where the component performs pixel to pixel comparison of the reports and the visual differences are highlighted.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">This Tableau regression component can be very useful for <span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/automate-tableau-testing/">automating the testing of Tableau reports</a></span> or Dashboards during in-place upgrades or changes.</span> </p> </div>
</div>
<div class="elementor-element elementor-element-687fa79 elementor-widget elementor-widget-image" data-id="687fa79" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1097" height="286" src="https://www.datagaps.com/wp-content/uploads/tableau-regression.png" class="attachment-full size-full wp-image-36084" alt="Tableau Regression Test" srcset="https://www.datagaps.com/wp-content/uploads/tableau-regression.png 1097w, https://www.datagaps.com/wp-content/uploads/tableau-regression-300x78.png 300w, https://www.datagaps.com/wp-content/uploads/tableau-regression-1024x267.png 1024w, https://www.datagaps.com/wp-content/uploads/tableau-regression-768x200.png 768w" sizes="(max-width: 1097px) 100vw, 1097px" /> </div>
</div>
<div class="elementor-element elementor-element-cf64761 elementor-widget elementor-widget-heading" data-id="cf64761" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Tableau Upgrade</h3> </div>
</div>
<div class="elementor-element elementor-element-77bb2a9 elementor-widget elementor-widget-text-editor" data-id="77bb2a9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW238833216 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW238833216 BCX0">Tableau Upgrade Component in BI validator helps in automated</span><span class="NormalTextRun SCXW238833216 BCX0"> report</span><span class="NormalTextRun SCXW238833216 BCX0"> testing by comparing the same or different reports of same or different Tableau sources.</span></span><span class="LineBreakBlob BlobObject DragDrop SCXW238833216 BCX0"><span class="SCXW238833216 BCX0"> </span><br class="SCXW238833216 BCX0" /></span><span class="LineBreakBlob BlobObject DragDrop SCXW238833216 BCX0"><span class="SCXW238833216 BCX0"> </span><br class="SCXW238833216 BCX0" /></span><span class="TextRun SCXW238833216 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW238833216 BCX0">There can be multiple versions of the same Tableau report, especially across different environments. This </span><span class="NormalTextRun SCXW238833216 BCX0">component</span><span class="NormalTextRun SCXW238833216 BCX0"> is useful for </span><span class="NormalTextRun SCXW238833216 BCX0">validating</span><span class="NormalTextRun SCXW238833216 BCX0"> these reports. For example, a report may differ between the production and testing environments.</span></span><span class="LineBreakBlob BlobObject DragDrop SCXW238833216 BCX0"><span class="SCXW238833216 BCX0"> </span><br class="SCXW238833216 BCX0" /></span><span class="LineBreakBlob BlobObject DragDrop SCXW238833216 BCX0"><span class="SCXW238833216 BCX0"> </span><br class="SCXW238833216 BCX0" /></span><span class="TextRun SCXW238833216 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW238833216 BCX0">The comparison is done in the same manner as regression testing where the differences between the reports can be pointed out both in terms of text as well as appearance.</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-74f2e1d elementor-widget elementor-widget-image" data-id="74f2e1d" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="620" height="239" src="https://www.datagaps.com/wp-content/uploads/upgrade-results.png" class="attachment-full size-full wp-image-36089" alt="" srcset="https://www.datagaps.com/wp-content/uploads/upgrade-results.png 620w, https://www.datagaps.com/wp-content/uploads/upgrade-results-300x116.png 300w" sizes="(max-width: 620px) 100vw, 620px" /> </div>
</div>
<div class="elementor-element elementor-element-3f59cf8 elementor-widget elementor-widget-heading" data-id="3f59cf8" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Generate BI DataFlows</h2> </div>
</div>
<div class="elementor-element elementor-element-a0651f0 elementor-widget elementor-widget-text-editor" data-id="a0651f0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">Generate BI DataFlows is a handy and convenient feature provided by <a href="https://www.datagaps.com/dataops-suite/"><span style="color: #0000ff;">Datagaps DataOps suite</span></a> to generate multiple dataflows at once for Business Intelligence components like Tableau.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">Generate BI DataFlows feature is beneficial in migration scenarios as it enables efficient data comparison between the original and migrated platforms and supports the validations like BI source, Regression and Upgrade.</span></p><p><span data-contrast="auto">By generating multiple dataflows based on selected reports, users can quickly detect discrepancies or inconsistencies that may arise during the migration process, ensuring data integrity and accuracy while minimizing potential errors.</span> <span data-contrast="auto">Furthermore, when dealing with a large volume of reports, this feature speeds up the validation process, minimizes manual effort, and improves overall efficiency in detecting and resolving inconsistencies.</span><span data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-983fe61 elementor-widget elementor-widget-text-editor" data-id="983fe61" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong><span class="TextRun SCXW142289448 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW142289448 BCX0">Here’s</span><span class="NormalTextRun SCXW142289448 BCX0"> how the BI Generate Dataflow wizard looks like:</span></span><span class="EOP SCXW142289448 BCX0" data-ccp-props="{}"> </span></strong></p> </div>
</div>
<div class="elementor-element elementor-element-e655eed elementor-widget elementor-widget-image" data-id="e655eed" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="584" height="317" src="https://www.datagaps.com/wp-content/uploads/BI-Generate-Dataflow-wizard-start.png" class="attachment-full size-full wp-image-36085" alt="" srcset="https://www.datagaps.com/wp-content/uploads/BI-Generate-Dataflow-wizard-start.png 584w, https://www.datagaps.com/wp-content/uploads/BI-Generate-Dataflow-wizard-start-300x163.png 300w" sizes="(max-width: 584px) 100vw, 584px" /> </div>
</div>
<div class="elementor-element elementor-element-fffbef7 elementor-widget elementor-widget-text-editor" data-id="fffbef7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">As seen from the image, the wizard starts by generating the Dataflow details. The connection details like the engine, validation type, Source-Data Source and Target-Data Source are to be provided by users.</span> </p> </div>
</div>
<div class="elementor-element elementor-element-f72b925 elementor-alert-info elementor-widget elementor-widget-alert" data-id="f72b925" data-element_type="widget" data-e-type="widget" data-widget_type="alert.default">
<div class="elementor-widget-container">
<div class="elementor-alert" role="alert">
<span class="elementor-alert-title">Note</span>
<span class="elementor-alert-description">BI source validation and Regression validation types do not prompt for Target-Data source </span>
<button type="button" class="elementor-alert-dismiss" aria-label="Dismiss this alert.">
<span aria-hidden="true">×</span>
</button>
</div>
</div>
</div>
<div class="elementor-element elementor-element-deb220f elementor-widget elementor-widget-text-editor" data-id="deb220f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW218298140 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW218298140 BCX0">Let’s</span><span class="NormalTextRun SCXW218298140 BCX0"> take a closer look at the steps involved in “<strong>Generate BI Dataflows</strong>”</span></span><span class="EOP SCXW218298140 BCX0" data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-c8cd4dd elementor-widget elementor-widget-heading" data-id="c8cd4dd" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Reports </h3> </div>
</div>
<div class="elementor-element elementor-element-9df8cd7 elementor-widget elementor-widget-text-editor" data-id="9df8cd7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">The Reports section prompts users to select pages from the required reports in the validation process. For Data Compare Validation and Upgrade Validation, both source and target pages will be required. For other cases, only the source page will be needed.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">The “</span><b><span data-contrast="auto">Edit Target</span></b><span data-contrast="auto">” wizard allows for manual addition of pages from the target report, providing flexibility and control over the validation scope.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">Here is a sample screenshot of the extraction of source and target pages from the source and target report respectively</span><span data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-919a1ee elementor-widget elementor-widget-image" data-id="919a1ee" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1722" height="773" src="https://www.datagaps.com/wp-content/uploads/BI-GENERATE-DATAFLOW_reports-selection.png" class="attachment-full size-full wp-image-36086" alt="Reports - Generate BI Dataflows" srcset="https://www.datagaps.com/wp-content/uploads/BI-GENERATE-DATAFLOW_reports-selection.png 1722w, https://www.datagaps.com/wp-content/uploads/BI-GENERATE-DATAFLOW_reports-selection-300x135.png 300w, https://www.datagaps.com/wp-content/uploads/BI-GENERATE-DATAFLOW_reports-selection-1024x460.png 1024w, https://www.datagaps.com/wp-content/uploads/BI-GENERATE-DATAFLOW_reports-selection-768x345.png 768w, https://www.datagaps.com/wp-content/uploads/BI-GENERATE-DATAFLOW_reports-selection-1536x690.png 1536w" sizes="(max-width: 1722px) 100vw, 1722px" /> </div>
</div>
<div class="elementor-element elementor-element-982b173 elementor-widget elementor-widget-heading" data-id="982b173" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Visual Mapping and Column Mapping (only in Data Compare Validation)</h3> </div>
</div>
<div class="elementor-element elementor-element-b3d2826 elementor-widget elementor-widget-text-editor" data-id="b3d2826" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">The “<strong>Visual Mapping</strong>” section allows users to load and compare source and target pages and then establish connections between corresponding tables.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">It consists of three sections namely Source Page, Target Page, and Mapping.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">In the source page and target page, respective Tableau worksheets are loaded and on selecting the worksheets option, users can preview the data.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">After loading the source and target pages,</span><span data-contrast="none"> in the mapping section</span><span data-contrast="none">, </span><span data-contrast="auto">the dataset columns of source and target will be automatically mapped for each mapping.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">After Visual Mapping, the “<strong>Column Mapping</strong>” section displays the columns of the source dataset and target dataset that were selected for the data comparison. It provides a count of the number of dataset columns that are mapped and unmapped in the “<strong>Mapped</strong>” and “<strong>Unmapped</strong>” tabs respectively.</span> </p> </div>
</div>
<div class="elementor-element elementor-element-81de601 elementor-widget elementor-widget-heading" data-id="81de601" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Filters (for the rest of the validation types)</h3> </div>
</div>
<div class="elementor-element elementor-element-d7364a6 elementor-widget elementor-widget-text-editor" data-id="d7364a6" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW149665980 BCX0">The filters section enables users to apply the filters and parameters </span><span class="NormalTextRun SCXW149665980 BCX0">on the reports to help in </span><span class="NormalTextRun SCXW149665980 BCX0">validating</span><span class="NormalTextRun SCXW149665980 BCX0"> them. These filters can either be applied and selected directly through reports or they can be parameterized as well.</span></p> </div>
</div>
<div class="elementor-element elementor-element-d565fee elementor-widget elementor-widget-heading" data-id="d565fee" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Options </h3> </div>
</div>
<div class="elementor-element elementor-element-1e2dceb elementor-widget elementor-widget-text-editor" data-id="1e2dceb" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">Options section varies depending on the type of validation selected by the user. Options section is the pre final stage of generating the flows where some of the advanced options and comparison options are prompted to be selected as per the liking of the user to get the results as they like.</span><span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">Here’s a sample screenshot of options section before generating the dataflows</span><span data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-68b8ace elementor-widget elementor-widget-image" data-id="68b8ace" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="490" height="305" src="https://www.datagaps.com/wp-content/uploads/options.png" class="attachment-full size-full wp-image-36087" alt="" srcset="https://www.datagaps.com/wp-content/uploads/options.png 490w, https://www.datagaps.com/wp-content/uploads/options-300x187.png 300w" sizes="(max-width: 490px) 100vw, 490px" /> <figcaption class="widget-image-caption wp-caption-text">This screenshot indicates report to report comparison options to be selected. </figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-5c91446 elementor-widget elementor-widget-heading" data-id="5c91446" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Generate </h3> </div>
</div>
<div class="elementor-element elementor-element-9d4da8b elementor-widget elementor-widget-text-editor" data-id="9d4da8b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW56390715 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW56390715 BCX0">Generate section helps to generate multiple dataflows with the selected type of validation depending on the number of selected workbooks for tableau.</span></span><span class="EOP SCXW56390715 BCX0" data-ccp-props="{}"> </span></p> </div>
</div>
<div class="elementor-element elementor-element-2e4df88 elementor-widget elementor-widget-image" data-id="2e4df88" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="560" height="250" src="https://www.datagaps.com/wp-content/uploads/BI-generate.png" class="attachment-full size-full wp-image-36088" alt="BI Generate BI DataFlows" srcset="https://www.datagaps.com/wp-content/uploads/BI-generate.png 560w, https://www.datagaps.com/wp-content/uploads/BI-generate-300x134.png 300w" sizes="(max-width: 560px) 100vw, 560px" /> </div>
</div>
<div class="elementor-element elementor-element-eee8019 elementor-widget elementor-widget-text-editor" data-id="eee8019" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW40040190 BCX0">The above screenshot </span><span class="NormalTextRun SCXW40040190 BCX0">indicates</span><span class="NormalTextRun SCXW40040190 BCX0"> that</span><span class="NormalTextRun SCXW40040190 BCX0"> four dataflows are set to be generated on clicking the Generate BI Dataflows button. These dataflows are the same type of validation (Tableau Regression Validation in this case)</span></p> </div>
</div>
<div class="elementor-element elementor-element-f6a9b87 elementor-widget elementor-widget-heading" data-id="f6a9b87" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Stress Test Plan </h2> </div>
</div>
<div class="elementor-element elementor-element-5a2646b elementor-widget elementor-widget-text-editor" data-id="5a2646b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span data-contrast="auto">To automate the stress testing and performance testing of Tableau Reports, <span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/bi-validator/">Datagaps DataOps suite BI Validator</a></span> comes with a component called Stress Test Plan to simulate the number of users actively accessing the reports to analyze how Tableau reports and dashboards perform under heavy load. </span></p><p>Results of the stress test plan can be used to point out performance issues, optimize data models and queries to ensure the robustness of the Tableau environment to handle heavy usage patterns.</p><p>Stress Test Plan allows users to perform the stress testing for multiple views from multiple workbooks at once enabling the flexibility and automation to check for performance bottlenecks of Tableau reports. <span data-ccp-props="{}"> </span></p><p><span data-contrast="auto">For more information on Stress Test Plan, check out “<span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/blog/tableau-performance-testing-spot-issues/">Tableau Performance Testing</a></span>”.</span><span data-ccp-props="{}"> </span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-f7822a4 e-flex e-con-boxed e-con e-parent" data-id="f7822a4" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-0652215 elementor-widget elementor-widget-heading" data-id="0652215" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Integration with CI/CD tools and Pipelines </h2> </div>
</div>
<div class="elementor-element elementor-element-2b7ff88 elementor-widget elementor-widget-text-editor" data-id="2b7ff88" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span lang="EN-IN" xml:lang="EN-IN" data-contrast="auto">Until now, we have seen the dataflows being automatically generated for the Tableau Reports Validation. But if a need arises for a dataflow to be a part of your daily pipeline or you want to run the dataflows based on certain deployments. The application comes with the flexibility to integrate with the CI/CD tools like GITHUB Actions or Jenkins. </span></p><p><span style="color: #444444;">BI Validator provides a command <span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/automate-tableau-testing/">line interface and REST API for kicking off test plans. </a></span></span>Customers can use this interface to execute tests automatically from the CI/CD tools. </p><p><span lang="EN-IN" xml:lang="EN-IN" data-contrast="auto">In addition to these features, DataOps Suite comes with other interesting features like application in built pipelines where the set of Tableau BI dataflows can be run automatically in a certain order either in sequence or parallel.<br /></span><br /><span class="TextRun SCXW231131234 BCX0" lang="EN-IN" xml:lang="EN-IN" data-contrast="auto"><span class="NormalTextRun SCXW231131234 BCX0">Also, </span><span class="NormalTextRun SCXW231131234 BCX0">there’s</span><span class="NormalTextRun SCXW231131234 BCX0"> an inbuilt scheduler in the application where the users can schedule the run of these pipelines involving these BI dataflows well in advance. The jobs can be scheduled to run once or repeatedly as well.</span></span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-a17d5ef e-flex e-con-boxed e-con e-parent" data-id="a17d5ef" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-5d4923e elementor-widget elementor-widget-text-editor" data-id="5d4923e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div style="font-family: 'Poppins', sans-serif; color: #444444; text-align: center;">
<h1 style="font-size: 32px; font-weight: bold; margin-bottom: 20px;">Achieve Seamless & Automated Tableau Report Validation</h1>
<!-- Subheading -->
<h2 style="font-size: 22px; margin-bottom: 30px;">Leverage the advanced capabilities of Datagaps DataOps Suite BI Validator for unmatched efficiency.</h2>
<!-- Call-to-Action -->
<a style="font-size: 20px; font-weight: bold; color: #ffffff; background-color: green; padding: 12px 25px; border-radius: 5px; text-decoration: none;" href="https://www.datagaps.com/data-ops-suite-trial-request/">Get Started Today</a>
</div> </div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/automating-tableau-reports-validation-the-easy-path-to-trusted-insights/">Automating Tableau Reports Validation: The Easy Path to Trusted Insights </a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
<wfw:commentRss>https://www.datagaps.com/blog/automating-tableau-reports-validation-the-easy-path-to-trusted-insights/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>DataOps Suite Accelerates CI/CD for Data Pipelines Through Testing Automation </title>
<link>https://www.datagaps.com/blog/dataops-suite-accelerates-ci-cd-for-data-pipelines-through-testing-automation/</link>
<comments>https://www.datagaps.com/blog/dataops-suite-accelerates-ci-cd-for-data-pipelines-through-testing-automation/#respond</comments>
<dc:creator><![CDATA[Eshaa Shah]]></dc:creator>
<pubDate>Thu, 09 May 2024 14:24:52 +0000</pubDate>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<guid isPermaLink="false">https://www.datagaps.com/?p=30291</guid>
<description><![CDATA[<p>DataOps Suite can help with your existing CI/CD pipelines and Accelerate CI/CD for Data Pipelines with Testing Automation</p>
<p>The post <a href="https://www.datagaps.com/blog/dataops-suite-accelerates-ci-cd-for-data-pipelines-through-testing-automation/">DataOps Suite Accelerates CI/CD for Data Pipelines Through Testing Automation </a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="30291" class="elementor elementor-30291" data-elementor-post-type="post">
<div class="elementor-element elementor-element-63ab17c e-flex e-con-boxed e-con e-parent" data-id="63ab17c" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-8ef96fa elementor-widget elementor-widget-text-editor" data-id="8ef96fa" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW148730300 BCX0">The </span><a href="https://www.datagaps.com/dataops-suite/"><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0" style="color: #0000ff;">DataOps</span></a><span class="NormalTextRun SCXW148730300 BCX0"><span style="color: #0000ff;"> Suite</span> is revolutionizing CI/CD pipelines for data and code validation. This suite enhances cloud development’s speed, quality, and reliability, promising a new era of efficiency for continuous integration and continuous deployment. </span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e9ed993 e-flex e-con-boxed e-con e-parent" data-id="e9ed993" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-727a41e elementor-widget elementor-widget-heading" data-id="727a41e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">What is DataOps?</h2> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-261c8cf e-flex e-con-boxed e-con e-parent" data-id="261c8cf" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-6f3a2d7 elementor-widget elementor-widget-text-editor" data-id="6f3a2d7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW148730300 BCX0">As Gartner defines it, “</span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> is a collaborative data management practice focused on improving the communication, integration, and automation of data flows between data managers and data consumers across an organization. </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> aims to deliver value faster by creating predictable delivery and change management of data, data models, and related artifacts.”</span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-969a77d e-flex e-con-boxed e-con e-parent" data-id="969a77d" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-8677b62 elementor-widget elementor-widget-heading" data-id="8677b62" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Advantages of Automated Testing in CI/CD Pipelines</h2> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-9bed384 e-flex e-con-boxed e-con e-parent" data-id="9bed384" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-188efdd elementor-widget elementor-widget-video" data-id="188efdd" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/youtu.be\/OIRtksLlcaM","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
<div class="elementor-element elementor-element-8939cc4 elementor-widget elementor-widget-text-editor" data-id="8939cc4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Prompt feedback </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span></span></strong><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><strong><span class="SCXW148730300 BCX0"> </span></strong><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automated testing can run continually and fast, providing prompt feedback when flaws are identified. <br></span></span></p>
<p><strong><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Improved collaboration </span></span></strong><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><strong><span class="SCXW148730300 BCX0"> </span></strong></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automating testing and deployment can </span><span class="NormalTextRun SCXW148730300 BCX0">assist</span><span class="NormalTextRun SCXW148730300 BCX0"> development, testing, and operations teams in working more efficiently and effectively, lowering the stake of errors and miscommunication. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Rapid deployment</strong> </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automated CI/CD pipelines can </span><span class="NormalTextRun SCXW148730300 BCX0">facilitate</span><span class="NormalTextRun SCXW148730300 BCX0"> companies to build, test, and deploy changes in code, data, or models faster than manually integrating changes. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><strong><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Improved MTTR </span></span></strong><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><strong><span class="SCXW148730300 BCX0"> </span></strong><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><a href="https://en.wikipedia.org/wiki/CI/CD" target="_blank">CI/CD</a> can support reducing the average time it takes to recover from a probable failure, measured by the MTTR. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><strong><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Transparency </span></span></strong><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automated data compliance can offer complete transparency by allowing authorized personnel to access updated compliance data immediately. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><strong><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Reduced manual effort </span></span></strong><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Test automation can reduce manual effort when the same test must be run repeatedly, freeing time for more rigorous manual tests. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><strong><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Boost data accuracy </span></span></strong><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><strong><span class="SCXW148730300 BCX0"> </span></strong><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automated tests can be more precise and cover better test cases. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Product consistency</strong> </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automated testing can help generate and compare </span><span class="NormalTextRun SCXW148730300 BCX0">numerous</span><span class="NormalTextRun SCXW148730300 BCX0"> test results, ensuring product consistency. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Faster delivery of high-quality software</strong> </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Automated GUI testing can help developers </span><span class="NormalTextRun SCXW148730300 BCX0">identify</span><span class="NormalTextRun SCXW148730300 BCX0"> and fix issues more quickly, reducing the time it takes to deliver high-quality software. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">Efficient Pipeline Management with </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> Suite Create a more reliable system for Data Pipeline with </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> Suite. </span></span></p>
<p><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">The </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> Suite offers end-to-end comprehensive data validation tools for both data and code pipelines: </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Dual Pipeline Orchestration:</strong> Simplifies complex data and code workflow efficiency across environments. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Advance Data Observability:</strong> Supervises data and code pipelines and proactively </span><span class="NormalTextRun SCXW148730300 BCX0">maintains</span><span class="NormalTextRun SCXW148730300 BCX0"> high quality and performance standards. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Deployment Automation:</strong> </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">Opts</span><span class="NormalTextRun SCXW148730300 BCX0"> CI/CD principles for effortless updates and integrations, decreasing downtime. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0"><strong>Testing Automation:</strong> Ensures data integrity and code functionality data validation. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><span class="SCXW148730300 BCX0"> </span><br class="SCXW148730300 BCX0"></span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-6ab646b e-flex e-con-boxed e-con e-parent" data-id="6ab646b" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-8382c24 elementor-widget elementor-widget-heading" data-id="8382c24" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Integration and Impact </h2> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-b2c43e7 e-flex e-con-boxed e-con e-parent" data-id="b2c43e7" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-2f1904a elementor-widget elementor-widget-text-editor" data-id="2f1904a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW148730300 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW148730300 BCX0">The </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> Suite provides easy-to-use connectivity for data sources and coding environments, thorough testing and validation by development and operational standards, and smooth transitions and upgrades seamlessly integrating into existing CI/CD frameworks. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW148730300 BCX0"><br class="SCXW148730300 BCX0" /></span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-80b8634 e-flex e-con-boxed e-con e-parent" data-id="80b8634" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-c8adc49 elementor-widget elementor-widget-heading" data-id="c8adc49" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Success Story </h2> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-791dad9 e-flex e-con-boxed e-con e-parent" data-id="791dad9" data-element_type="container" data-e-type="container">
<div class="e-con-inner">
<div class="elementor-element elementor-element-175518e elementor-widget elementor-widget-text-editor" data-id="175518e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="NormalTextRun SCXW148730300 BCX0">A renowned tech company </span><span class="NormalTextRun SCXW148730300 BCX0">leveraged</span><span class="NormalTextRun SCXW148730300 BCX0"> the </span><span class="NormalTextRun SpellingErrorV2Themed SCXW148730300 BCX0">DataOps</span><span class="NormalTextRun SCXW148730300 BCX0"> Suite to reconcile its data and code pipelines, resulting in a 40% decrease in deployment cycles and a 30% improvement in data and code quality.</span></p> </div>
</div>
<div class="elementor-element elementor-element-b901ee9 elementor-widget elementor-widget-text-editor" data-id="b901ee9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW191101251 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW191101251 BCX0" data-ccp-charstyle="Heading 3 Char"><strong>Conclusion</strong> </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW191101251 BCX0"><span class="SCXW191101251 BCX0"> </span><br class="SCXW191101251 BCX0" /></span><span class="LineBreakBlob BlobObject DragDrop SCXW191101251 BCX0"><span class="SCXW191101251 BCX0"> </span><br class="SCXW191101251 BCX0" /></span><span class="TextRun SCXW191101251 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW191101251 BCX0"><a href="https://www.datagaps.com/dataops-suite/"><span style="color: #0000ff;"><strong>DataOps Suite</strong></span></a> can help with CI/CD to easily streamline the overall cloud development process and </span><span class="NormalTextRun SCXW191101251 BCX0">maintain</span><span class="NormalTextRun SCXW191101251 BCX0"> high data quality and reliability. </span><span class="NormalTextRun SCXW191101251 BCX0">DataGaps</span><span class="NormalTextRun SCXW191101251 BCX0"> recently went live on </span><span class="NormalTextRun SCXW191101251 BCX0">a webinar</span><span class="NormalTextRun SCXW191101251 BCX0"> sharing “Accelerate CI/CD for Data Pipelines with Testing Automation” in which we shared how our Industry Agnostic </span><span class="NormalTextRun SpellingErrorV2Themed SCXW191101251 BCX0">D</span><span class="NormalTextRun SpellingErrorV2Themed SCXW191101251 BCX0">ataOps</span> <span class="NormalTextRun SCXW191101251 BCX0">Suite aids in accelerating data pipelines within CI/CD. They automate repetitive tasks, bring teams together, and give you a clear view of what is happening. This makes your data pipelines run smoother and faster and, most importantly, delivers high-quality data you can trust for your applications and reports. </span></span><span class="LineBreakBlob BlobObject DragDrop SCXW191101251 BCX0"><span class="SCXW191101251 BCX0"> </span><br class="SCXW191101251 BCX0" /></span><span class="LineBreakBlob BlobObject DragDrop SCXW191101251 BCX0"><span class="SCXW191101251 BCX0"> </span><br class="SCXW191101251 BCX0" /></span><span class="TextRun SCXW191101251 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW191101251 BCX0">In CI/CD data pipelines, catching bugs early mimics the benefits of early bug detection in software development. It alleviates pressure on data engineers, fosters higher code quality, prevents downstream issues, and allows for quicker resolution, </span><span class="NormalTextRun SCXW191101251 BCX0">u</span><span class="NormalTextRun SCXW191101251 BCX0">ltimately building</span> <span class="NormalTextRun SCXW191101251 BCX0">a solid foundation for reliable data flow.</span></span><span class="EOP SCXW191101251 BCX0" data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335557856":16777215,"335559738":0,"335559739":240,"335559740":279}"> </span></p> </div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-b5d01a8 e-flex e-con-boxed e-con e-parent" data-id="b5d01a8" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-b587334 e-flex e-con-boxed e-con e-child" data-id="b587334" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-0aa78b0 elementor-widget elementor-widget-text-editor" data-id="0aa78b0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: left;"><span class="TextRun SCXW184731117 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW184731117 BCX0">Hidden Defects in your Data Pipeline?</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-24abb13 elementor-widget elementor-widget-text-editor" data-id="24abb13" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span class="TextRun SCXW184731117 BCX0" lang="EN-US" xml:lang="EN-US" data-contrast="none"><span class="NormalTextRun SCXW184731117 BCX0">Looking for Reduced Downstream Impacts?</span></span></p> </div>
</div>
<div class="elementor-element elementor-element-cd63170 elementor-widget elementor-widget-video" data-id="cd63170" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/youtu.be\/OIRtksLlcaM","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/dataops-suite-accelerates-ci-cd-for-data-pipelines-through-testing-automation/">DataOps Suite Accelerates CI/CD for Data Pipelines Through Testing Automation </a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
<wfw:commentRss>https://www.datagaps.com/blog/dataops-suite-accelerates-ci-cd-for-data-pipelines-through-testing-automation/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>The Integration of Data Build Tool (dbt) and Datagaps DataOps Suite</title>
<link>https://www.datagaps.com/blog/the-integration-of-data-build-tool-dbt-and-datagaps-dataops-suite/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Thu, 04 Jan 2024 05:28:56 +0000</pubDate>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=25000</guid>
<description><![CDATA[<p>Data Warehouses provide a comprehensive view of data across many different sources, and proper analysis can encourage better business decisions and problem-solving. However, building and maintaining an effective data warehouse requires careful thought and consideration</p>
<p>The post <a href="https://www.datagaps.com/blog/the-integration-of-data-build-tool-dbt-and-datagaps-dataops-suite/">The Integration of Data Build Tool (dbt) and Datagaps DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="25000" class="elementor elementor-25000" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-c266c1a elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="c266c1a" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f89d2c9" data-id="f89d2c9" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0321b78 elementor-widget elementor-widget-heading" data-id="0321b78" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">What is dbt Data Build Tool?</h4> </div>
</div>
<div class="elementor-element elementor-element-a64546f elementor-widget elementor-widget-text-editor" data-id="a64546f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Introduction – In today’s data-driven world, organizations are continually seeking ways to optimize data operations, enhance data quality, and ensure robust governance practices. The integration of powerful tools like DBT (Data Build Tool) and Datagaps DataOps Suite has emerged as a game-changing solution. Individually, these tools offer impressive capabilities, and when integrated, they create a symbiotic relationship that addresses challenges related to data transformation, quality, profiling, and observability. In this article, we delve into the integration of DBT and Datagaps DataOps Suite, exploring how this combination fosters more efficient data operations and empowers data-driven decision-making.</p> </div>
</div>
<div class="elementor-element elementor-element-109e5b9 elementor-widget elementor-widget-heading" data-id="109e5b9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Data Build Tool(dbt): A Brief Overview
</h4> </div>
</div>
<div class="elementor-element elementor-element-55d9339 elementor-widget elementor-widget-text-editor" data-id="55d9339" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DBT, or <a href="https://en.wikipedia.org/wiki/Data_build_tool">Data Build Tool</a>, is a popular open-source command-line tool designed primarily for transforming data analytics. It allows data analysts and engineers to transform data within their warehouse in a structured and version-controlled manner. With its focus on SQL-based transformations, DBT promotes collaboration, transparency, and maintainability in data pipelines.</p> </div>
</div>
<div class="elementor-element elementor-element-b537fff elementor-widget elementor-widget-heading" data-id="b537fff" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Datagaps DataOps Suite: Enhancing Data Quality and Governance</h4> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-ebfc1ea elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="ebfc1ea" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-968226c" data-id="968226c" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-05ec210 elementor-widget elementor-widget-image" data-id="05ec210" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="212" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.webp" class="attachment-large size-large wp-image-25004" alt="DataOps-Suite-Logo-clr" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.webp 885w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-300x99.webp 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-768x254.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-784d531" data-id="784d531" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d60042f elementor-widget elementor-widget-text-editor" data-id="d60042f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
DataOps suite that encompasses a range of functionalities, including data quality assurance, profiling, observability, and governance. The suite’s core objective is to improve data operations across the board by ensuring that data is of high quality, well understood, and observable throughout its lifecycle. With plug-and-play integration systems, a plethora of supported data sources, cloud and on-prem system support, and much more the Suite is most comprehensive Data Quality, Profiling, and Governance system in the data sphere. </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-14817f5 elementor-widget elementor-widget-heading" data-id="14817f5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Integration Mechanics: Leveraging REST APIs and Plugin Support
</h4> </div>
</div>
<div class="elementor-element elementor-element-06b2180 elementor-widget elementor-widget-image" data-id="06b2180" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1402" height="860" src="https://www.datagaps.com/wp-content/uploads/REST-APIs-and-plugin-support.jpeg" class="attachment-full size-full wp-image-25037" alt="" srcset="https://www.datagaps.com/wp-content/uploads/REST-APIs-and-plugin-support.jpeg 1402w, https://www.datagaps.com/wp-content/uploads/REST-APIs-and-plugin-support-300x184.jpeg 300w, https://www.datagaps.com/wp-content/uploads/REST-APIs-and-plugin-support-1024x628.jpeg 1024w, https://www.datagaps.com/wp-content/uploads/REST-APIs-and-plugin-support-768x471.jpeg 768w" sizes="(max-width: 1402px) 100vw, 1402px" /> </div>
</div>
<div class="elementor-element elementor-element-81ab9a0 elementor-widget elementor-widget-text-editor" data-id="81ab9a0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The integration of DBT and Datagaps DataOps Suite is made possible through a combination of REST APIs and plugin support within the Datagaps DataOps Suite. This allows seamless communication between the two platforms, enabling data engineers and analysts to utilize the strengths of both tools without friction. Data transformations orchestrated through DBT can be automatically monitored, profiled, and governed within the Datagaps DataOps Suite, forming a cohesive data management ecosystem.</p> </div>
</div>
<div class="elementor-element elementor-element-75c77e8 elementor-widget elementor-widget-heading" data-id="75c77e8" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try DataOps Suite – Free Trial</a></h4> </div>
</div>
<div class="elementor-element elementor-element-e5253f1 elementor-widget elementor-widget-heading" data-id="e5253f1" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Plugins for Integration
</h4> </div>
</div>
<div class="elementor-element elementor-element-3dbed41 elementor-widget elementor-widget-text-editor" data-id="3dbed41" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The suite provides end-users with complete customizability options and developer-friendly tools to create any set of customizations, integrations, processing nodes, test case validation, and other toolsets.</p> </div>
</div>
<div class="elementor-element elementor-element-7b80b15 elementor-widget elementor-widget-image" data-id="7b80b15" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1408" height="832" src="https://www.datagaps.com/wp-content/uploads/Plugins-for-Integration.jpeg" class="attachment-full size-full wp-image-25048" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Plugins-for-Integration.jpeg 1408w, https://www.datagaps.com/wp-content/uploads/Plugins-for-Integration-300x177.jpeg 300w, https://www.datagaps.com/wp-content/uploads/Plugins-for-Integration-1024x605.jpeg 1024w, https://www.datagaps.com/wp-content/uploads/Plugins-for-Integration-768x454.jpeg 768w" sizes="(max-width: 1408px) 100vw, 1408px" /> </div>
</div>
<div class="elementor-element elementor-element-671c788 elementor-widget elementor-widget-text-editor" data-id="671c788" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
As seen in the images, the application will help developers define parameters, variables, and datasets which are subsequently used in the code component to create toolsets capable of a variety of tasks. </div>
</div>
<div class="elementor-element elementor-element-06ab5b0 elementor-widget elementor-widget-text-editor" data-id="06ab5b0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The end users only have to plug in the required variables or parameters and the application subsequently takes care of the translations. The example showcases the various types of plugins that themselves can be exported or imported with the different use-cases seen in the enterprise data space. These plugins can be build in Scala as well.</p> </div>
</div>
<div class="elementor-element elementor-element-32ddf2c elementor-widget elementor-widget-heading" data-id="32ddf2c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Trigger a DBT Job and Compare the Results </h4> </div>
</div>
<div class="elementor-element elementor-element-94d2b10 elementor-widget elementor-widget-text-editor" data-id="94d2b10" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In this showcase, we use the plugin showcased earlier to first trigger a DBT job after which we compare two datasets for incorrectness. </div>
</div>
<div class="elementor-element elementor-element-b2ca97d elementor-widget elementor-widget-image" data-id="b2ca97d" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1793" height="958" src="https://www.datagaps.com/wp-content/uploads/Trigger-a-DBT-Job-and-Compare-the-Results-Dataflow.jpeg" class="attachment-full size-full wp-image-25053" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Trigger-a-DBT-Job-and-Compare-the-Results-Dataflow.jpeg 1793w, https://www.datagaps.com/wp-content/uploads/Trigger-a-DBT-Job-and-Compare-the-Results-Dataflow-300x160.jpeg 300w, https://www.datagaps.com/wp-content/uploads/Trigger-a-DBT-Job-and-Compare-the-Results-Dataflow-1024x547.jpeg 1024w, https://www.datagaps.com/wp-content/uploads/Trigger-a-DBT-Job-and-Compare-the-Results-Dataflow-768x410.jpeg 768w, https://www.datagaps.com/wp-content/uploads/Trigger-a-DBT-Job-and-Compare-the-Results-Dataflow-1536x821.jpeg 1536w" sizes="(max-width: 1793px) 100vw, 1793px" /> </div>
</div>
<div class="elementor-element elementor-element-30e320e elementor-widget elementor-widget-text-editor" data-id="30e320e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The dataflow displays the plugin node, loading of the 2 datasets, and data comparison node.</p> </div>
</div>
<div class="elementor-element elementor-element-4a966ec elementor-widget elementor-widget-image" data-id="4a966ec" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1377" height="522" src="https://www.datagaps.com/wp-content/uploads/Options-of-Chosen-Plugin.jpeg" class="attachment-full size-full wp-image-25058" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Options-of-Chosen-Plugin.jpeg 1377w, https://www.datagaps.com/wp-content/uploads/Options-of-Chosen-Plugin-300x114.jpeg 300w, https://www.datagaps.com/wp-content/uploads/Options-of-Chosen-Plugin-1024x388.jpeg 1024w, https://www.datagaps.com/wp-content/uploads/Options-of-Chosen-Plugin-768x291.jpeg 768w" sizes="(max-width: 1377px) 100vw, 1377px" /> </div>
</div>
<div class="elementor-element elementor-element-966cf08 elementor-widget elementor-widget-text-editor" data-id="966cf08" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Options of Chosen Plugin
</div>
</div>
<div class="elementor-element elementor-element-db2c813 elementor-widget elementor-widget-image" data-id="db2c813" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1363" height="844" src="https://www.datagaps.com/wp-content/uploads/Output-Plugin.jpeg" class="attachment-full size-full wp-image-25064" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Output-Plugin.jpeg 1363w, https://www.datagaps.com/wp-content/uploads/Output-Plugin-300x186.jpeg 300w, https://www.datagaps.com/wp-content/uploads/Output-Plugin-1024x634.jpeg 1024w, https://www.datagaps.com/wp-content/uploads/Output-Plugin-768x476.jpeg 768w" sizes="(max-width: 1363px) 100vw, 1363px" /> </div>
</div>
<div class="elementor-element elementor-element-2eef2b2 elementor-widget elementor-widget-text-editor" data-id="2eef2b2" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Output Plugin </div>
</div>
<div class="elementor-element elementor-element-2a36a90 elementor-widget elementor-widget-image" data-id="2a36a90" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1404" height="874" src="https://www.datagaps.com/wp-content/uploads/Output-of-Data-Compare.jpeg" class="attachment-full size-full wp-image-25065" alt="" srcset="https://www.datagaps.com/wp-content/uploads/Output-of-Data-Compare.jpeg 1404w, https://www.datagaps.com/wp-content/uploads/Output-of-Data-Compare-300x187.jpeg 300w, https://www.datagaps.com/wp-content/uploads/Output-of-Data-Compare-1024x637.jpeg 1024w, https://www.datagaps.com/wp-content/uploads/Output-of-Data-Compare-768x478.jpeg 768w" sizes="(max-width: 1404px) 100vw, 1404px" /> </div>
</div>
<div class="elementor-element elementor-element-babfcc0 elementor-widget elementor-widget-text-editor" data-id="babfcc0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Output of Data Compare
</div>
</div>
<div class="elementor-element elementor-element-2b7b039 elementor-widget elementor-widget-text-editor" data-id="2b7b039" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>As seen in the screenshots, the application first triggers a DBT job based on the user’s inputs, then loads up the Source and Target Datasets which were created post the job’s completion, and finally runs a data comparison check.</p><p>The failure of this test case triggers a notification to the users showcasing the various mismatched, isolated and duplicate records seen in the datasets as a compiled report.</p> </div>
</div>
<div class="elementor-element elementor-element-ffc5abb elementor-widget elementor-widget-heading" data-id="ffc5abb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Benefits and Use-Cases
</h4> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-689d3f5 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="689d3f5" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ba8a7ee" data-id="ba8a7ee" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-7da8015 elementor-widget elementor-widget-image" data-id="7da8015" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="1" src="https://www.datagaps.com/wp-content/uploads/Enhanced-Data-Quality-and-Profiling-Benefits.svg" class="attachment-medium_large size-medium_large wp-image-25179" alt="" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-a3fadac" data-id="a3fadac" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4bfc252 elementor-widget elementor-widget-text-editor" data-id="4bfc252" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Enhanced Data Quality and Profiling:</strong> The integration ensures that data transformed using DBT undergoes rigorous data quality checks and profiling within the Datagaps DataOps Suite. This leads to cleaner, more reliable data, reducing errors and enhancing the trustworthiness of analyses.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-348d635 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="348d635" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0643243" data-id="0643243" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-43a7dee elementor-widget elementor-widget-image" data-id="43a7dee" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="1" src="https://www.datagaps.com/wp-content/uploads/Observability-and-Monitoring-Benefits.svg" class="attachment-medium_large size-medium_large wp-image-25180" alt="" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-daaf553" data-id="daaf553" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-59d87b8 elementor-widget elementor-widget-text-editor" data-id="59d87b8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Observability and Monitoring:</strong> Datagaps DataOps Suite’s <a href="https://www.datagaps.com/dataops-suite/dataflow/">observability features</a> allow teams to monitor data transformations executed through DBT in real-time. This enables swift identification of issues, performance bottlenecks, and anomalies, leading to quicker resolution times.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-d8b0956 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="d8b0956" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-257d60d" data-id="257d60d" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-04e675f elementor-widget elementor-widget-image" data-id="04e675f" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="1" src="https://www.datagaps.com/wp-content/uploads/Efficient-Collaboration-Benefits.svg" class="attachment-medium_large size-medium_large wp-image-25181" alt="" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0c56c0e" data-id="0c56c0e" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-a46fc97 elementor-widget elementor-widget-text-editor" data-id="a46fc97" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Efficient Collaboration:</strong> Data engineers and analysts can collaborate more efficiently using the integrated solution. DBT’s transformation logic and Datagaps DataOps Suite’s observability tools provide a shared context for better communication and decision-making.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-d9c249a elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="d9c249a" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-d3efdc7" data-id="d3efdc7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-c53b4d9 elementor-widget elementor-widget-image" data-id="c53b4d9" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="1" src="https://www.datagaps.com/wp-content/uploads/Unified-Data-Governance-Benefits.svg" class="attachment-medium_large size-medium_large wp-image-25182" alt="" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-01def82" data-id="01def82" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0da4879 elementor-widget elementor-widget-text-editor" data-id="0da4879" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Unified Data Governance:</strong> While Datagaps DataOps Suite doesn’t focus primarily on governance, its capabilities contribute to effective data governance practices. Organizations can ensure that data transformations adhere to compliance requirements and maintain a clear understanding of data lineage.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-c91534f elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="c91534f" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-5d95cb6" data-id="5d95cb6" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cd949cb elementor-widget elementor-widget-image" data-id="cd949cb" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="1" src="https://www.datagaps.com/wp-content/uploads/Automated-Documentation-Benefits.svg" class="attachment-medium_large size-medium_large wp-image-25183" alt="" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-8190662" data-id="8190662" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4e1bec0 elementor-widget elementor-widget-text-editor" data-id="4e1bec0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Automated Documentation:</strong> The integration automates the process of documenting data transformations, lineage, and quality checks. This documentation is vital for maintaining a historical record of changes and ensuring transparency.</p> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-bbb5050 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="bbb5050" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-763a2f5" data-id="763a2f5" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-ad1220c elementor-widget elementor-widget-image" data-id="ad1220c" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="1" src="https://www.datagaps.com/wp-content/uploads/Holistic-Data-Strategy-Benefits.svg" class="attachment-medium_large size-medium_large wp-image-25199" alt="" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-c29f0e3" data-id="c29f0e3" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-a876f98 elementor-widget elementor-widget-text-editor" data-id="a876f98" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>Holistic Data Strategy:</strong> Integrating DBT and Datagaps DataOps Suite supports organizations in developing a holistic data strategy. It bridges the gap between transformation and operational aspects, empowering organizations to make data-driven decisions confidently.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-87c7155 elementor-widget elementor-widget-heading" data-id="87c7155" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Conclusion</h4> </div>
</div>
<div class="elementor-element elementor-element-aa9c3f4 elementor-widget elementor-widget-text-editor" data-id="aa9c3f4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The integration of Data Build Tool (DBT) and Datagaps DataOps Suite is a remarkable example of how combining specialized tools can create a more robust and comprehensive data management solution. By harnessing the strengths of both tools, organizations can streamline data transformation, enhance data quality, improve observability, and ensure better data governance practices.</p> </div>
</div>
<div class="elementor-element elementor-element-efdcfce elementor-widget elementor-widget-text-editor" data-id="efdcfce" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
As the data landscape continues to evolve, this integration paves the way for more effective and efficient data operations, enabling organizations to unlock the full potential of their data-driven initiatives. </div>
</div>
<div class="elementor-element elementor-element-3f5fd4c elementor-widget elementor-widget-heading" data-id="3f5fd4c" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try DataOps Suite – Free Trial</a></h4> </div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/the-integration-of-data-build-tool-dbt-and-datagaps-dataops-suite/">The Integration of Data Build Tool (dbt) and Datagaps DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Data Profiling In Pharma Datasets Using DataOps Suite</title>
<link>https://www.datagaps.com/blog/data-profiling-in-pharma-datasets-using-dataops-suite/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 14 Feb 2023 13:26:37 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=11758</guid>
<description><![CDATA[<p>Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions.</p>
<p>The post <a href="https://www.datagaps.com/blog/data-profiling-in-pharma-datasets-using-dataops-suite/">Data Profiling In Pharma Datasets Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="11758" class="elementor elementor-11758" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-9a46834 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="9a46834" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions. Data profiling involves examining and summarizing the characteristics of a dataset in order to identify patterns, trends, and anomalies in the data. By tracking aggregations and patterns in the data, it is possible to identify potential issues or anomalies that may need to be addressed in order to improve the quality of the data. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions. Data profiling involves examining and summarizing the characteristics of a dataset in order to identify patterns, trends, and anomalies in the data. By tracking aggregations and patterns in the data, it is possible to identify potential issues or anomalies that may need to be addressed in order to improve the quality of the data.</p> </div>
</div>
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Pattern Recognition and Tracking of Keys and Strings</h5> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the pharmaceutical industry, it is common for different vendors to provide datasets that contain information on the same subjects or entities. For example, a vendor may provide a dataset containing information on clinical trial participants, while another vendor may provide a dataset containing information on patient outcomes.</p><p> </p><p>In order to accurately merge or join these datasets, it is important that the primary keys used to identify the subjects or entities are consistent. For example, if one dataset uses a 9-digit numerical key to identify participants, it is important that any other datasets that contain information on the same participants also use a 9-digit numerical key.</p><p> </p><p>If the pattern of the primary keys is not consistent, it can make it difficult or impossible to accurately link records from different datasets. This can lead to errors or incorrect analyses and can compromise the overall integrity of the data.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-0db8f78 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0db8f78" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-1d20bf4" data-id="1d20bf4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-791af0e elementor-widget elementor-widget-image" data-id="791af0e" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="212" src="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.png" class="attachment-large size-large wp-image-11756" alt="DataOps-Suite-Logo-1" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1.png 885w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-300x99.png 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Suite-Logo-1-768x254.png 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-018c404" data-id="018c404" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d3fe57d elementor-widget elementor-widget-text-editor" data-id="d3fe57d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
To ensure the consistency of primary keys in pharma datasets, it is important to regularly monitor the patterns of primary keys and identify any potential issues. The DataOps Suite’s profile tracking node?? can be used to monitor the patterns of primary keys and alert you to any inconsistencies. This can help you ensure the quality and integrity of your pharma datasets and avoid any potential issues that could arise from inconsistent primary keys. </div>
</div>
<div class="elementor-element elementor-element-8c11b6b elementor-widget elementor-widget-spacer" data-id="8c11b6b" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-4bfe18a elementor-widget elementor-widget-text-editor" data-id="4bfe18a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
As seen in the example below, originally the only pattern seen in the datasets was a 9-digit key. However, in the latest run post, an update from the client we see a new alphanumeric pattern is also seen in the system. This might indicate a data-type change and a definite notification in data governance. </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-e4db7c6 elementor-widget elementor-widget-image" data-id="e4db7c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="1374" height="373" src="https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key.png" class="attachment-1536x1536 size-1536x1536 wp-image-11759" alt="data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key" srcset="https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key.png 1374w, https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key-300x81.png 300w, https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key-1024x278.png 1024w, https://www.datagaps.com/wp-content/uploads/data-profile-node-result-showcasing-a-change-in-the-patterns-of-a-primary-key-768x208.png 768w" sizes="(max-width: 1374px) 100vw, 1374px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data profile node result showcasing a change in the patterns of a primary key</figcaption>
</figure>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-deb51bb elementor-widget elementor-widget-heading" data-id="deb51bb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Outliers in Patient Claims and Drug Sales Datasets</h5> </div>
</div>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Outliers are values in a dataset that are significantly different from the majority of the other values. In inpatient claims and drug sales datasets, outliers can occur in various aggregates, such as averages, standard deviations, minimum values, and maximum values. </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Outliers can have a significant impact on the results of any analyses or modeling efforts, as they can distort the overall patterns or trends in the data. For example, if a dataset contains an outlier value that is significantly higher or lower than the majority of the other values, it could skew the average or standard deviation, leading to incorrect or misleading results.</p><p> </p><p><a href="https://www.datagaps.com/etl-testing-tools/etl-validator-download/">Try DataOps Suite – Free Trial</a></p> </div>
</div>
<div class="elementor-element elementor-element-324bb87 elementor-widget elementor-widget-text-editor" data-id="324bb87" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<ul><li>A few examples of how variations in min-max values and standard deviations can help identify anomalies in patient claims and drug sales datasets:<br />If the minimum value for a dataset decreases significantly over time, it could indicate an anomaly or error in the data. For example, if the minimum value for a column containing drug prices decreases significantly from one month to the next, it could indicate that the price was entered incorrectly or that the drug is being sold at a significantly discounted rate.</li><li>If the maximum value for a dataset increases significantly over time, it could also indicate an anomaly or error in the data. For example, if the maximum value for a column containing drug prices increases significantly from one month to the next, it could indicate that the price was entered incorrectly or that the drug is being sold at a significantly inflated rate.</li><li>If the standard deviation for a dataset increases significantly over time, it could also indicate an anomaly or error in the data. For example, if the standard deviation for a column containing drug prices increases significantly from one month to the next, it could indicate that the prices are becoming more variable than expected, which could be a sign of an anomaly or error.</li></ul> </div>
</div>
<div class="elementor-element elementor-element-62f0ab3 elementor-widget elementor-widget-image" data-id="62f0ab3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<figure class="wp-caption">
<img loading="lazy" decoding="async" width="877" height="454" src="https://www.datagaps.com/wp-content/uploads/data-profile-node-results.png" class="attachment-1536x1536 size-1536x1536 wp-image-11761" alt="data-profile-node-results" srcset="https://www.datagaps.com/wp-content/uploads/data-profile-node-results.png 877w, https://www.datagaps.com/wp-content/uploads/data-profile-node-results-300x155.png 300w, https://www.datagaps.com/wp-content/uploads/data-profile-node-results-768x398.png 768w" sizes="(max-width: 877px) 100vw, 877px" /> <figcaption class="widget-image-caption wp-caption-text">DataOps Suite: Data Profile Node Results</figcaption>
</figure>
</div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Also Read: <a href="https://www.datagaps.com/blog/data-drift-using-dataops-data-profiling/">Data Drift Using DataOps Data Profiling</a></p> </div>
</div>
<div class="elementor-element elementor-element-98c53c5 elementor-widget elementor-widget-heading" data-id="98c53c5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Distributions and List of Values Deltas</h5> </div>
</div>
<div class="elementor-element elementor-element-2955a70 elementor-widget elementor-widget-text-editor" data-id="2955a70" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
For inpatient claims and drug sales datasets, it is important to monitor the distribution of values across different columns and variables. The DataOps Suite’s profile node can provide various plots and statistics that can help you understand the distribution of values in your data. </div>
</div>
<div class="elementor-element elementor-element-fc1846b elementor-widget elementor-widget-text-editor" data-id="fc1846b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
For example, if you are analyzing a dataset containing information on patient claims, you might be interested in the distribution of diagnoses across different diagnosis codes. The profile node can provide a histogram or other plot showing the distribution of diagnosis codes, which can help you identify any patterns or trends in the data. </div>
</div>
<div class="elementor-element elementor-element-a757e12 elementor-widget elementor-widget-text-editor" data-id="a757e12" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In addition to monitoring the distribution of values, it can also be useful to monitor a list of values (LOV) deltas. LOV deltas refer to the difference between the list of values used in one dataset and the list of values used in another dataset. For example, if you are comparing a dataset of patient claims from one year to a dataset of patient claims from the previous year, you might be interested in the LOV deltas between the two datasets. </div>
</div>
<div class="elementor-element elementor-element-33aa808 elementor-blockquote--skin-border elementor-widget elementor-widget-blockquote" data-id="33aa808" data-element_type="widget" data-e-type="widget" data-widget_type="blockquote.default">
<div class="elementor-widget-container">
<blockquote class="elementor-blockquote">
<p class="elementor-blockquote__content">
The DataOps Suite’s profile node can provide statistics on LOV deltas, which can help you identify any changes in the list of values used in your datasets. This can be useful for ensuring the quality and consistency of your data, and for identifying any potential issues or discrepancies. </p>
</blockquote>
</div>
</div>
<div class="elementor-element elementor-element-e5f9257 elementor-widget elementor-widget-text-editor" data-id="e5f9257" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><strong>As seen below 2 examples:</strong></p><p> </p><p><strong>Example A</strong> deals with showcasing a change in the number of distinct values seen in a geography key of a patient claims dataset.</p><p> </p><p><strong>Example B</strong> showcases how the distribution of sales among different “Lines of Therapy” has been drastically changed indicating either an issue in the calculation of LOT, a change in behavior of the LOT in the drug in question, or worse a bug in the ETL.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-2e1dce7 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="2e1dce7" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-b54c24b" data-id="b54c24b" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8539833 elementor-widget elementor-widget-image" data-id="8539833" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="876" height="336" src="https://www.datagaps.com/wp-content/uploads/Example-A.png" class="attachment-1536x1536 size-1536x1536 wp-image-11763" alt="Example-A" srcset="https://www.datagaps.com/wp-content/uploads/Example-A.png 876w, https://www.datagaps.com/wp-content/uploads/Example-A-300x115.png 300w, https://www.datagaps.com/wp-content/uploads/Example-A-768x295.png 768w" sizes="(max-width: 876px) 100vw, 876px" /> </div>
</div>
<div class="elementor-element elementor-element-fca6f9b elementor-widget elementor-widget-image" data-id="fca6f9b" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="884" height="240" src="https://www.datagaps.com/wp-content/uploads/Example-A-01.png" class="attachment-1536x1536 size-1536x1536 wp-image-11764" alt="Example-A-01" srcset="https://www.datagaps.com/wp-content/uploads/Example-A-01.png 884w, https://www.datagaps.com/wp-content/uploads/Example-A-01-300x81.png 300w, https://www.datagaps.com/wp-content/uploads/Example-A-01-768x209.png 768w" sizes="(max-width: 884px) 100vw, 884px" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-cd01ff0" data-id="cd01ff0" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-9f41a73 elementor-widget elementor-widget-image" data-id="9f41a73" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="953" height="660" src="https://www.datagaps.com/wp-content/uploads/Example-B.png" class="attachment-1536x1536 size-1536x1536 wp-image-11762" alt="Example-B" srcset="https://www.datagaps.com/wp-content/uploads/Example-B.png 953w, https://www.datagaps.com/wp-content/uploads/Example-B-300x208.png 300w, https://www.datagaps.com/wp-content/uploads/Example-B-768x532.png 768w" sizes="(max-width: 953px) 100vw, 953px" /> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-c2e0aa5 elementor-widget elementor-widget-heading" data-id="c2e0aa5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-80ad1bd elementor-widget elementor-widget-text-editor" data-id="80ad1bd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In conclusion, data profiling is an important step in the data preparation process, and it is especially important in the pharmaceutical industry where data quality and integrity are critical. The DataOps Suite’s profile node is a powerful tool that can help you perform data profiling on your pharma datasets, and it can provide valuable insights and help you identify any potential issues or inconsistencies.</p> </div>
</div>
<div class="elementor-element elementor-element-980d9ed elementor-widget elementor-widget-text-editor" data-id="980d9ed" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Some of the key features of the profile <span class="fontColorRed">node</span> include overview statistics, column statistics, and column distribution plots, which can all be useful in understanding the contents, structure, and quality of your data. In addition, the profile node can help you identify anomalies and outliers in your data, and it can provide statistics on LOV deltas, which can be useful for ensuring the consistency of your data.</p> </div>
</div>
<div class="elementor-element elementor-element-b9d0214 elementor-widget elementor-widget-text-editor" data-id="b9d0214" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Overall, the DataOps Suite’s profile node is a valuable tool that can help you ensure the quality and integrity of your pharma datasets and support more accurate and reliable analyses and modeling efforts.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-c60bdbc e-flex e-con-boxed e-con e-parent" data-id="c60bdbc" data-element_type="container" data-e-type="container" data-settings="{"background_background":"classic"}">
<div class="e-con-inner">
<div class="elementor-element elementor-element-d8d5665 e-con-full e-flex e-con e-child" data-id="d8d5665" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-4cab2cb elementor-widget elementor-widget-heading" data-id="4cab2cb" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Get a Free POC scheduled today!</a></h2> </div>
</div>
</div>
<div class="elementor-element elementor-element-f4cf0ce e-con-full e-flex e-con e-child" data-id="f4cf0ce" data-element_type="container" data-e-type="container">
<div class="elementor-element elementor-element-d547dfe elementor-align-right elementor-widget elementor-widget-button" data-id="d547dfe" data-element_type="widget" data-e-type="widget" data-widget_type="button.default">
<div class="elementor-widget-container">
<div class="elementor-button-wrapper">
<a class="elementor-button elementor-button-link elementor-size-md" href="https://www.datagaps.com/request-demo/">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Click here</span>
</span>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>The post <a href="https://www.datagaps.com/blog/data-profiling-in-pharma-datasets-using-dataops-suite/">Data Profiling In Pharma Datasets Using DataOps Suite</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Monitoring Your Data Pipelines In Production – DataOps</title>
<link>https://www.datagaps.com/blog/monitoring-your-etl-test-data-pipelines-in-production-dataops-suite/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Fri, 13 Jan 2023 14:29:53 +0000</pubDate>
<category><![CDATA[Data Quality]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=9971</guid>
<description><![CDATA[<p>DataOps is managing data in its raw form or transforming it from one format to another. It ensures that the data flowing through your production pipelines is accurate and reliable. DataOps combines software development and infrastructure management best practices with traditional</p>
<p>The post <a href="https://www.datagaps.com/blog/monitoring-your-etl-test-data-pipelines-in-production-dataops-suite/">Monitoring Your Data Pipelines In Production – DataOps</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="9971" class="elementor elementor-9971" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-306567d elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="306567d" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-9cc1db2" data-id="9cc1db2" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-2d420d3 elementor-widget elementor-widget-text-editor" data-id="2d420d3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>DataOps is managing data in its raw form or transforming it from one format to another. It ensures that the data flowing through your production pipelines is accurate and reliable. DataOps combines software development and infrastructure management best practices with traditional data management to optimize data flow throughout the organization, improving the data quality used for decision-making, analytics, and reporting.</p> </div>
</div>
<div class="elementor-element elementor-element-fc57b57 elementor-widget elementor-widget-text-editor" data-id="fc57b57" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The data you produce will touch multiple endpoints across the enterprise, and it is necessary to ensure that you create stable and accurate processes in your ETL testing data pipelines. Monitoring these pipelines continuously and identifying any potential issues as they arise is essential, and there are some basic steps all teams can implement to monitor them more effectively. In this post, I will describe some of the activities you should perform to ensure that you deliver high-quality data to your end users.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-62e898d elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="62e898d" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-no">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-8ec3758" data-id="8ec3758" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-b0b56ab elementor-widget elementor-widget-text-editor" data-id="b0b56ab" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<h2>How To Monitor Key Activities?</h2><p>Key monitoring activities will vary based on the environment, data, and ETL data testing pipeline tools. How organizations measure the effectiveness of their overall solution will depend on their particular use case. But although the ETL testing pipeline may look different from one organization to another, the approach used to monitor the streams are often quite similar.</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-556c9ab" data-id="556c9ab" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-74f24a8 elementor-widget elementor-widget-image" data-id="74f24a8" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="427" src="https://www.datagaps.com/wp-content/uploads/Monitoring-scaled-1-1024x683.webp" class="attachment-large size-large wp-image-9973" alt="Monitoring-scaled" srcset="https://www.datagaps.com/wp-content/uploads/Monitoring-scaled-1-1024x683.webp 1024w, https://www.datagaps.com/wp-content/uploads/Monitoring-scaled-1-300x200.webp 300w, https://www.datagaps.com/wp-content/uploads/Monitoring-scaled-1-768x512.webp 768w, https://www.datagaps.com/wp-content/uploads/Monitoring-scaled-1-1536x1024.webp 1536w, https://www.datagaps.com/wp-content/uploads/Monitoring-scaled-1-2048x1366.webp 2048w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-02250fb elementor-widget elementor-widget-text-editor" data-id="02250fb" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Below are a few recommendations that ensure data teams can get the maximum valuable insight from their monitoring strategy.</p><p>Also Read: <a href="https://www.datagaps.com/blog/data-drift-using-dataops-data-profiling/">Data Drift Using DataOps Data Profiling</a></p> </div>
</div>
<div class="elementor-element elementor-element-0a7d059 elementor-widget elementor-widget-accordion" data-id="0a7d059" data-element_type="widget" data-e-type="widget" data-widget_type="accordion.default">
<div class="elementor-widget-container">
<div class="elementor-accordion">
<div class="elementor-accordion-item">
<div id="elementor-tab-title-1091" class="elementor-tab-title" data-tab="1" role="button" aria-controls="elementor-tab-content-1091" aria-expanded="false">
<span class="elementor-accordion-icon elementor-accordion-icon-right" aria-hidden="true">
<span class="elementor-accordion-icon-closed"><i class="fas fa-plus"></i></span>
<span class="elementor-accordion-icon-opened"><i class="fas fa-minus"></i></span>
</span>
<a class="elementor-accordion-title" tabindex="0">Create and Maintain A Data Entry</a>
</div>
<div id="elementor-tab-content-1091" class="elementor-tab-content elementor-clearfix" data-tab="1" role="region" aria-labelledby="elementor-tab-title-1091"><p>A data inventory lists the organization’s data sources and their relationships. It includes details such as the location of the data, the metadata used to store it, transformation rules that align with business purposes, and any data-related processes. Business owners often interchangeably refer to a “data inventory” as a “data dictionary” without recognizing that there is a subtle difference. A data dictionary describes the structure of a given set of data in terms of fields, types, values, relationships, and other descriptors. In contrast, a data inventory provides a more comprehensive view of all the different types of data stored by the organization and the ways it can be combined and used to inform business decisions. The broader view of a data inventory helps the enterprise keep track of its data and understand how it is used in different departments or applications within the organization.<br />Compiling the data inventory to use in an ETL process up-front makes it easier to track how information is used within the organization and identify any issues that could impact the data quality. This first step is essential as it is the foundation upon which the entire organization will ensure the quality and delivery of its monitoring. It also serves as the single point of truth when different stakeholders provide their objective interpretation of how the business rules should be applied across departments. Building a data inventory manually can be resource-intensive and time-consuming, but the right tool that automates aspects of the collection can make this process easier.</p></div>
</div>
<div class="elementor-accordion-item">
<div id="elementor-tab-title-1092" class="elementor-tab-title" data-tab="2" role="button" aria-controls="elementor-tab-content-1092" aria-expanded="false">
<span class="elementor-accordion-icon elementor-accordion-icon-right" aria-hidden="true">
<span class="elementor-accordion-icon-closed"><i class="fas fa-plus"></i></span>
<span class="elementor-accordion-icon-opened"><i class="fas fa-minus"></i></span>
</span>
<a class="elementor-accordion-title" tabindex="0">Identify Key Stages in the Data Pipeline</a>
</div>
<div id="elementor-tab-content-1092" class="elementor-tab-content elementor-clearfix" data-tab="2" role="region" aria-labelledby="elementor-tab-title-1092"><p>Depending on your organization’s infrastructure, you may have several different ETL data pipelines feeding into your production environments with independent data preparation, transformation rules, and batch processing results. Understanding and documenting the stages in your ETL pipeline and the inputs and outputs within each phase will promote accurate identification when the data reaches the end user. Establishing clear policies and standards for data governance—that is, managing the availability, usability, integrity, and security of the data—will ensure end-to-end visibility across your enterprise. Proper data governance will identify errors that can be fixed quickly before they can disrupt operations. One challenge to remember is that as the number of users grows and the number of systems increases, these pipelines become increasingly complicated to manage and track. However, several ETL Testing tools in the market can simplify this process. For example, DataGaps’ ETL Validator is an end-to-end automated testing tool that enables developers to validate their complicated ETL flows and improve overall data transformation, quality, and loading before data is used within production-ready business processes. Using it will highlight errors in your pipelines long before they become a problem.</p></div>
</div>
<div class="elementor-accordion-item">
<div id="elementor-tab-title-1093" class="elementor-tab-title" data-tab="3" role="button" aria-controls="elementor-tab-content-1093" aria-expanded="false">
<span class="elementor-accordion-icon elementor-accordion-icon-right" aria-hidden="true">
<span class="elementor-accordion-icon-closed"><i class="fas fa-plus"></i></span>
<span class="elementor-accordion-icon-opened"><i class="fas fa-minus"></i></span>
</span>
<a class="elementor-accordion-title" tabindex="0">Analyze Pipeline Metrics and Monitoring Logs</a>
</div>
<div id="elementor-tab-content-1093" class="elementor-tab-content elementor-clearfix" data-tab="3" role="region" aria-labelledby="elementor-tab-title-1093"><p>At each stage of the pipeline, you should consider monitoring several metrics, such as latency, throughput, error rates, and storage consumption. It helps to ensure that the pipeline performs as expected. Finding the bottlenecks that occur along the way will help you optimize the performance of your pipeline and make better use of your resources. Additionally, set up alerts that can notify you whenever a particular metric exceeds a specified threshold so that you can take corrective action before the issue causes downtime. The biggest obstacle you will face is knowing the specific metrics to monitor, as different data sources could have any number of candidate outputs. You especially want to be careful when building workflows that monitor metrics and automate results-based tasks. For example, you may decide to monitor the compute resource consumption of pipelines and commit autoscaling of your servers if the pipelines reach a threshold. In this scenario, your autoscaling costs could skyrocket if you fail to identify and monitor avoidable bottlenecks in each pipeline. Again, tools such as ETL Validator can help streamline the pipeline’s performance bottlenecks by reporting out-of-the-box and customized key performance metrics and helping you decide what steps to take under which scenario.</p></div>
</div>
<div class="elementor-accordion-item">
<div id="elementor-tab-title-1094" class="elementor-tab-title" data-tab="4" role="button" aria-controls="elementor-tab-content-1094" aria-expanded="false">
<span class="elementor-accordion-icon elementor-accordion-icon-right" aria-hidden="true">
<span class="elementor-accordion-icon-closed"><i class="fas fa-plus"></i></span>
<span class="elementor-accordion-icon-opened"><i class="fas fa-minus"></i></span>
</span>
<a class="elementor-accordion-title" tabindex="0">Dedicated Tools To Capture and Monitor Pipeline Metrics</a>
</div>
<div id="elementor-tab-content-1094" class="elementor-tab-content elementor-clearfix" data-tab="4" role="region" aria-labelledby="elementor-tab-title-1094"><p>There is an old saying: “give a person a chainsaw, and everything becomes a block of wood.” When dealing with ETL pipelines, developers often want to use a single application to monitor and resolve every potential error across the enterprise. Try to avoid this: The choice of application you select to scan your ETL pipeline needs to vary by the Data Governance principles of availability, usability, integrity, and security, and you should be open to multiple tools that cover each. For example, suppose you are performing complex data analytics. In that case, you may be better off using a purpose-built solution for advanced analytics capabilities focusing mainly on integrity and usability. More straightforward data migration and reconciliation tasks may emphasize monitoring and logging tools focusing chiefly on availability and security. Of course, you want to keep the number of solutions reasonable, as each new tool requires a learning curve that can become confusing and challenging to maintain separately. Under these circumstances, you can keep your tools separate while investing in a centralized dashboard to track all your performance indicators in a single location. This will allow you to keep an overview of the health of your pipeline at a glance, supporting your Data Governance maturity while simultaneously allowing you to incorporate new supporting applications over time.</p></div>
</div>
<div class="elementor-accordion-item">
<div id="elementor-tab-title-1095" class="elementor-tab-title" data-tab="5" role="button" aria-controls="elementor-tab-content-1095" aria-expanded="false">
<span class="elementor-accordion-icon elementor-accordion-icon-right" aria-hidden="true">
<span class="elementor-accordion-icon-closed"><i class="fas fa-plus"></i></span>
<span class="elementor-accordion-icon-opened"><i class="fas fa-minus"></i></span>
</span>
<a class="elementor-accordion-title" tabindex="0">Automate Your Data Collection</a>
</div>
<div id="elementor-tab-content-1095" class="elementor-tab-content elementor-clearfix" data-tab="5" role="region" aria-labelledby="elementor-tab-title-1095"><p>Performing regular data snapshots at regular intervals will allow you to periodically validate the accuracy of your data and ensure that it remains up-to-date. These snapshots will also help you to identify any recent changes that may require manual intervention to ensure that they are accounted for in the pipeline. Collecting data from disparate sources can be time-consuming, so you should look for a solution to automate this process. This will make it easier for you to keep your data current and prevent you from manually re-entering the data later, an experience your developers and engineers are bound to dislike.</p></div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-2681630 elementor-widget elementor-widget-text-editor" data-id="2681630" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The list of activities is not comprehensive, but it is an excellent start to making your ETL data pipelines more efficient and predictable. If you follow these guidelines, you should be able to improve your operational efficiency and minimize the risk of errors that can derail your data and lead to unnecessary delays.</p> </div>
</div>
<div class="elementor-element elementor-element-7e9190a elementor-widget elementor-widget-text-editor" data-id="7e9190a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>If you are curious about a suite of tools focused on <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/"><span style="color: #0000ff;">ETL Testing</span></a> and Data Quality Monitoring, We recommend DataGaps <span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/dataops-suite/">DataOps suite of tools</a></span>. With it, you can check the quality of the data you are ingesting, identify hard-to-find anomalies, perform data checks, build complex queries through a drop-and-drag interface, and receive real-time alerts as successes and failures occur. DataGaps also provides extensible plug-in components which allow developers to create custom-defined methods and promote code reuse for those hard-to-pinpoint use cases. If you are looking for a single provider to monitor your data pipelines in production, then <span style="color: #0000ff;"><a style="color: #0000ff;" href="https://www.datagaps.com/request-a-demo/" target="_blank" rel="noopener">request a demo today</a>.</span> Good luck.</p> </div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/monitoring-your-etl-test-data-pipelines-in-production-dataops-suite/">Monitoring Your Data Pipelines In Production – DataOps</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Validation of Salesforce Objects, Uploads and Updates</title>
<link>https://www.datagaps.com/blog/validation-of-salesforce-objects-uploads-and-updates/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Thu, 29 Dec 2022 16:34:31 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Data Validation]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=6967</guid>
<description><![CDATA[<p>Salesforce is a cloud-based CRM platform that helps businesses manage and analyze customer interactions and data throughout the customer lifecycle. It is used to store and organize information about customers, such as their contact details, communication history, and preferences.</p>
<p>The post <a href="https://www.datagaps.com/blog/validation-of-salesforce-objects-uploads-and-updates/">Validation of Salesforce Objects, Uploads and Updates</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="6967" class="elementor elementor-6967" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-f3ee7ad elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="f3ee7ad" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c68b878" data-id="c68b878" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-09034a1 elementor-widget elementor-widget-heading" data-id="09034a1" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Understanding Validation of Salesforce Objects, Uploads, <br> Updates and How Datagaps DataOps Can Help</h4> </div>
</div>
<div class="elementor-element elementor-element-379fe59 elementor-widget elementor-widget-heading" data-id="379fe59" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Intro to Salesforce</h5> </div>
</div>
<div class="elementor-element elementor-element-0345ed3 elementor-widget elementor-widget-text-editor" data-id="0345ed3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://www.salesforce.com/">Salesforce</a> is a cloud-based CRM platform that helps businesses manage and analyze customer interactions and data throughout the customer lifecycle. It is used to store and organize information about customers, such as their contact details, communication history, and preferences. This data is used to provide a complete view of the customer, which can help businesses understand their needs and personalize their interactions with them.</p> </div>
</div>
<div class="elementor-element elementor-element-2e621a3 elementor-widget elementor-widget-text-editor" data-id="2e621a3" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Salesforce also provides tools for managing customer relationships, including customer segmentation, lead management, and customer service. These tools help businesses to identify and prioritize their most valuable customers, track and follow up on leads, and provide timely and efficient customer service.</p> </div>
</div>
<div class="elementor-element elementor-element-665b149 elementor-widget elementor-widget-text-editor" data-id="665b149" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In addition to its CRM capabilities, Salesforce also provides a range of tools and features for data management and integration, including data import and export, data modeling, and data governance. This makes it possible for businesses to manage and analyze their customer data in a centralized location and to integrate it with other systems and applications.</p> </div>
</div>
<div class="elementor-element elementor-element-7c221c6 elementor-widget elementor-widget-heading" data-id="7c221c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Production vs Development – Metadata Validation</h5> </div>
</div>
<div class="elementor-element elementor-element-7e56b2c elementor-widget elementor-widget-text-editor" data-id="7e56b2c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div><p>In a Salesforce deployment, it is common for there to be differences in metadata between the development environment (also known as a sandbox or dev environment) and the production environment (also known as prod). This is because the development environment is often used for testing and experimentation, which can result in changes to the metadata that are not intended for the production environment.</p></div><p>Some examples of metadata changes that may occur in the development environment but not be intended for the production environment include:</p> </div>
</div>
<div class="elementor-element elementor-element-665dd37 elementor-widget elementor-widget-spacer" data-id="665dd37" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-0db8f78 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0db8f78" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-1d20bf4" data-id="1d20bf4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-3eccc05 elementor-view-default elementor-widget elementor-widget-icon" data-id="3eccc05" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-copy"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-018c404" data-id="018c404" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d3fe57d elementor-widget elementor-widget-text-editor" data-id="d3fe57d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Modifying the structure of objects or fields:</strong> This could involve adding or deleting fields, or changing field data types. For example, a developer may be testing a new feature that requires adding a new field to the Account object to store additional data. If this field is not needed in the production environment, it would be important to remove it before deploying the changes to prod. </div>
</div>
<div class="elementor-element elementor-element-8c11b6b elementor-widget elementor-widget-spacer" data-id="8c11b6b" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-6168d47 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6168d47" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-4b67854" data-id="4b67854" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-174db67 elementor-view-default elementor-widget elementor-widget-icon" data-id="174db67" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-file-alt"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-dfe3010" data-id="dfe3010" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d95bda2 elementor-widget elementor-widget-text-editor" data-id="d95bda2" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Changing page layouts or field-level security settings:</strong> This could involve modifying the layout of a page to display new fields or rearranging existing ones, or changing the visibility of fields based on user roles or profiles. For example, a business user may be testing a new page layout for the Account object in the dev environment, but this layout may not be ready for production. </div>
</div>
<div class="elementor-element elementor-element-c435a58 elementor-widget elementor-widget-spacer" data-id="c435a58" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-b765395 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="b765395" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-036622e" data-id="036622e" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-00c1e76 elementor-view-default elementor-widget elementor-widget-icon" data-id="00c1e76" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-share-alt"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ab56cf5" data-id="ab56cf5" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-78203fa elementor-widget elementor-widget-text-editor" data-id="78203fa" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Modifying workflow rules or approval processes:</strong> This could involve adding or modifying rules that trigger actions based on certain conditions, or changing the steps or participants in an approval process. For example, a developer may be testing a new workflow rule in the dev environment that sends an email notification when an Account is created, but this rule may not be ready for production. </div>
</div>
<div class="elementor-element elementor-element-433f832 elementor-widget elementor-widget-spacer" data-id="433f832" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-9f808f4 bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="9f808f4" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0e3983a" data-id="0e3983a" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-230c376 elementor-view-default elementor-widget elementor-widget-icon" data-id="230c376" data-element_type="widget" data-e-type="widget" data-widget_type="icon.default">
<div class="elementor-widget-container">
<div class="elementor-icon-wrapper">
<div class="elementor-icon">
<i aria-hidden="true" class="fas fa-folder-plus"></i> </div>
</div>
</div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-4dbb62c" data-id="4dbb62c" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4365065 elementor-widget elementor-widget-text-editor" data-id="4365065" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<strong>Adding or modifying custom objects or custom fields:</strong> This could involve creating new objects to store custom data or adding new fields to existing objects. For example, a business user may be testing a new custom object in the dev environment to track project tasks, but this object may not be ready for production. </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-b49a196 elementor-widget elementor-widget-spacer" data-id="b49a196" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-5d549c7 elementor-widget elementor-widget-text-editor" data-id="5d549c7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
These changes may occur in the development environment for a variety of reasons. For example, a developer may be testing a new feature or functionality and need to make changes to the metadata to support it. Or, a business user may be exploring different options for customizing the Salesforce instance and may make a series of changes as they iterate on their design. </div>
</div>
<div class="elementor-element elementor-element-4e68b86 elementor-widget elementor-widget-text-editor" data-id="4e68b86" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Here, the Metadata Validation Node of the <a href="https://www.datagaps.com/dataops-suite/dataflow/"> DataOps Suite.</a> can be used to ensure the metadata of the Dev and Prod Objects are in sync</p> </div>
</div>
<div class="elementor-element elementor-element-3080ee4 elementor-pagination-position-inside elementor-widget elementor-widget-image-carousel" data-id="3080ee4" data-element_type="widget" data-e-type="widget" data-settings="{"slides_to_show":"1","lazyload":"yes","autoplay":"no","navigation":"dots","infinite":"yes","effect":"slide","speed":500}" data-widget_type="image-carousel.default">
<div class="elementor-widget-container">
<div class="elementor-image-carousel-wrapper swiper" role="region" aria-roledescription="carousel" aria-label="Image Carousel" dir="ltr">
<div class="elementor-image-carousel swiper-wrapper" aria-live="polite">
<div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="1 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="2 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-02-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-02" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="3 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-03-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-03" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="4 of 4"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-04-1024x576.webp" alt="Metadata-Validation-Node-against-Dev-and-Prod-Salesforce-Schemas-04" /><div class="swiper-lazy-preloader"></div></figure></div> </div>
<div class="swiper-pagination"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-819725f elementor-widget elementor-widget-text-editor" data-id="819725f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A Metadata Validation Node against Dev and Prod Salesforce Schemas</p> </div>
</div>
<div class="elementor-element elementor-element-98c53c5 elementor-widget elementor-widget-heading" data-id="98c53c5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Upload Validation</h5> </div>
</div>
<div class="elementor-element elementor-element-f3a859d elementor-widget elementor-widget-text-editor" data-id="f3a859d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
There are several issues that can arise while uploading data to Salesforce, and the specific issues you may encounter can depend on the type of data you are uploading, the source of the data, and the type of Salesforce object you are uploading to. Here are a few examples of issues that can arise while uploading data to Salesforce: </div>
</div>
<div class="elementor-element elementor-element-a94c7b8 elementor-tabs-view-horizontal elementor-widget elementor-widget-tabs" data-id="a94c7b8" data-element_type="widget" data-e-type="widget" data-widget_type="tabs.default">
<div class="elementor-widget-container">
<div class="elementor-tabs">
<div class="elementor-tabs-wrapper" role="tablist" >
<div id="elementor-tab-title-1771" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="true" data-tab="1" role="tab" tabindex="0" aria-controls="elementor-tab-content-1771" aria-expanded="false">Data Formatting</div>
<div id="elementor-tab-title-1772" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="false" data-tab="2" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1772" aria-expanded="false">Data Integrity Issues</div>
<div id="elementor-tab-title-1773" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="false" data-tab="3" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1773" aria-expanded="false">Object-Specific Issues</div>
<div id="elementor-tab-title-1774" class="elementor-tab-title elementor-tab-desktop-title" aria-selected="false" data-tab="4" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1774" aria-expanded="false">Permission Issues</div>
</div>
<div class="elementor-tabs-content-wrapper" role="tablist" aria-orientation="vertical">
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="true" data-tab="1" role="tab" tabindex="0" aria-controls="elementor-tab-content-1771" aria-expanded="false">Data Formatting</div>
<div id="elementor-tab-content-1771" class="elementor-tab-content elementor-clearfix" data-tab="1" role="tabpanel" aria-labelledby="elementor-tab-title-1771" tabindex="0" hidden="false">If the data you are uploading is not properly formatted, it may not be accepted by Salesforce. For example, if you are uploading a CSV file and the data in the file is not properly structured, Salesforce may not be able to parse the data correctly.</div>
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="false" data-tab="2" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1772" aria-expanded="false">Data Integrity Issues</div>
<div id="elementor-tab-content-1772" class="elementor-tab-content elementor-clearfix" data-tab="2" role="tabpanel" aria-labelledby="elementor-tab-title-1772" tabindex="0" hidden="hidden">If the data you are uploading contains errors or inconsistencies, it may cause issues with the integrity of your Salesforce data. For example, if you are uploading a list of leads and some of the leads are missing required fields, the upload may fail.</div>
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="false" data-tab="3" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1773" aria-expanded="false">Object-Specific Issues</div>
<div id="elementor-tab-content-1773" class="elementor-tab-content elementor-clearfix" data-tab="3" role="tabpanel" aria-labelledby="elementor-tab-title-1773" tabindex="0" hidden="hidden">Each Salesforce object has its own set of fields and requirements, and if the data you are uploading does not meet these requirements, the upload may fail. For example, if you are uploading data to the Account object and the data does not contain a value for the required “Name” field, the upload may fail.</div>
<div class="elementor-tab-title elementor-tab-mobile-title" aria-selected="false" data-tab="4" role="tab" tabindex="-1" aria-controls="elementor-tab-content-1774" aria-expanded="false">Permission Issues</div>
<div id="elementor-tab-content-1774" class="elementor-tab-content elementor-clearfix" data-tab="4" role="tabpanel" aria-labelledby="elementor-tab-title-1774" tabindex="0" hidden="hidden">If you do not have the correct permissions in Salesforce, you may not be able to upload data to certain objects or fields. For example, if you are trying to upload data to a custom object that you do not have permission to access, the upload may fail.</div>
</div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-e5f9257 elementor-widget elementor-widget-text-editor" data-id="e5f9257" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The Data Compare Node can be used to pull data directly from the Salesforce Object and compare it against the file or dataset on premise as seen below. </div>
</div>
<div class="elementor-element elementor-element-9f0ffba elementor-pagination-position-inside elementor-widget elementor-widget-image-carousel" data-id="9f0ffba" data-element_type="widget" data-e-type="widget" data-settings="{"slides_to_show":"1","lazyload":"yes","autoplay":"no","navigation":"dots","infinite":"yes","effect":"slide","speed":500}" data-widget_type="image-carousel.default">
<div class="elementor-widget-container">
<div class="elementor-image-carousel-wrapper swiper" role="region" aria-roledescription="carousel" aria-label="Image Carousel" dir="ltr">
<div class="elementor-image-carousel swiper-wrapper swiper-image-stretch" aria-live="polite">
<div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="1 of 3"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Basic-Data-Compare-checking-salesforce-against-a-dataset-01.webp" alt="Basic-Data-Compare-checking-salesforce-against-a-dataset-01" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="2 of 3"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Basic-Data-Compare-checking-salesforce-against-a-dataset-02.webp" alt="Basic-Data-Compare-checking-salesforce-against-a-dataset-02" /><div class="swiper-lazy-preloader"></div></figure></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="3 of 3"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Basic-Data-Compare-checking-salesforce-against-a-dataset-03.webp" alt="Basic-Data-Compare-checking-salesforce-against-a-dataset-03" /><div class="swiper-lazy-preloader"></div></figure></div> </div>
<div class="swiper-pagination"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-195954c elementor-widget elementor-widget-text-editor" data-id="195954c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A Basic Data Compare checking salesforce against a dataset</p> </div>
</div>
<div class="elementor-element elementor-element-73d3e1e elementor-widget elementor-widget-heading" data-id="73d3e1e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Upsert vs Update</h5> </div>
</div>
<div class="elementor-element elementor-element-80ad1bd elementor-widget elementor-widget-text-editor" data-id="80ad1bd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
It is possible to encounter issues when uploading a large set of records that contain duplicates or need to be updated rather than inserted as new records. </div>
</div>
<div class="elementor-element elementor-element-980d9ed elementor-widget elementor-widget-text-editor" data-id="980d9ed" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
If you are using the upsert function and the records you are uploading contain duplicates based on the external ID field, the upsert function will treat these as updates rather than inserts and will update the existing records with the new data. This can be problematic if you want to insert the records as new records rather than updating the existing ones. </div>
</div>
<div class="elementor-element elementor-element-b9d0214 elementor-widget elementor-widget-text-editor" data-id="b9d0214" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In this case, you may want to use the update function instead of the upsert function. The update function allows you to specify a query to select the records you want to modify, rather than relying on the external ID field to identify matching records. This can be useful if you want to update records based on criteria other than the external ID, or if you want to insert records as new records rather than updating existing ones. </div>
</div>
<div class="elementor-element elementor-element-ab5bcb1 elementor-widget elementor-widget-text-editor" data-id="ab5bcb1" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
There are several issues that can arise while uploading data to Salesforce, and the specific issues you may encounter can depend on the type of data you are uploading, the source of the data, and the type of Salesforce object you are uploading to. Here are a few examples of issues that can arise while uploading data to Salesforce: </div>
</div>
<div class="elementor-element elementor-element-4c5ccfd elementor-widget elementor-widget-text-editor" data-id="4c5ccfd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
There are several issues that can arise while uploading data to Salesforce, and the specific issues you may encounter can depend on the type of data you are uploading, the source of the data, and the type of Salesforce object you are uploading to. Here are a few examples of issues that can arise while uploading data to Salesforce: </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-869ac3e elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="869ac3e" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-3c6189e" data-id="3c6189e" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-3fcfc10 elementor-widget elementor-widget-heading" data-id="3fcfc10" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Upsert</h5> </div>
</div>
<div class="elementor-element elementor-element-0684480 elementor-widget elementor-widget-text-editor" data-id="0684480" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The “upsert” function is used to either update existing records or create new records in an object, depending on whether a matching record already exists. When using the upsert function, you specify a field in the object that will be used as the unique identifier, called the “external ID”. If a record with a matching external ID already exists, the upsert function will update that record with the new data. If no matching record is found, the upsert function will create a new record with the provided data. </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-45cb62f" data-id="45cb62f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-4e25c01 elementor-widget elementor-widget-heading" data-id="4e25c01" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Update</h5> </div>
</div>
<div class="elementor-element elementor-element-6905a2c elementor-widget elementor-widget-text-editor" data-id="6905a2c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The “update” function is used to modify existing records in an object. When using the update function, you specify the records that you want to update using a query, and then specify the new field values that you want to set for those records. The update function will only modify records that already exist in the object, and will not create new records. </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-405c1ec elementor-widget elementor-widget-spacer" data-id="405c1ec" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-4f5cecc elementor-widget elementor-widget-text-editor" data-id="4f5cecc" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Both the upsert and update functions can be used to modify a single record or multiple records at once. They can be useful for updating or creating records in bulk, or for keeping data in Salesforce synchronized with data from other sources. </div>
</div>
<div class="elementor-element elementor-element-d0a9e7b elementor-widget elementor-widget-text-editor" data-id="d0a9e7b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
To solve the complex job of figuring out which records to use with Upsert and which to use Upload, the Data Compare Node comes in handy once again. Every node in the DataOps Suite has its results, and comparisons saved as views that can be called upon internally. This allows for reference and loops of the system to ensure complex solutions can be easily defined and solved in the Suite. </div>
</div>
<div class="elementor-element elementor-element-2a0c016 elementor-widget elementor-widget-text-editor" data-id="2a0c016" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In this case, post defining the Data Compare between the Salesforce Object and the CSV file in question the node creates a set of related views such as records present only in the salesforce object, records present only in the CSV file, the records marked as the differences, and more. Our focus will the dataset which is the difference and the dataset that houses records only in the CSV file. The dataset that houses records only in the CSV file has records that are not present in the salesforce object and thus needs the upsert functions to create the new records in the salesforce object. The other dataset marks the records present in the salesforce object that need to be updated in specific value sets. Here, the update function is used with a prefixed python code to identify the exact values to be updated. </div>
</div>
<div class="elementor-element elementor-element-4bb76f6 elementor-pagination-position-inside elementor-arrows-position-inside elementor-widget elementor-widget-image-carousel" data-id="4bb76f6" data-element_type="widget" data-e-type="widget" data-settings="{"slides_to_show":"1","lazyload":"yes","autoplay":"no","navigation":"both","infinite":"yes","effect":"slide","speed":500}" data-widget_type="image-carousel.default">
<div class="elementor-widget-container">
<div class="elementor-image-carousel-wrapper swiper" role="region" aria-roledescription="carousel" aria-label="Image Carousel" dir="ltr">
<div class="elementor-image-carousel swiper-wrapper" aria-live="polite">
<div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="1 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2MCwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTAxLndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="2 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2MiwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTAyLndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-02" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="3 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2MywidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTAzLndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-03" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="4 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2NCwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA0LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-04" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="5 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2NSwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA1LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-05" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="6 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2NywidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA2LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-06" /><div class="swiper-lazy-preloader"></div></figure></a></div><div class="swiper-slide" role="group" aria-roledescription="slide" aria-label="7 of 7"><a data-elementor-open-lightbox="yes" data-elementor-lightbox-slideshow="4bb76f6" data-elementor-lightbox-title="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07" data-e-action-hash="#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6NTk2OCwidXJsIjoiaHR0cHM6XC9cL3d3dy5kYXRhZ2Fwcy5jb21cL3dwLWNvbnRlbnRcL3VwbG9hZHNcL0RpZmZlcmVudC12aWV3cy1jcmVhdGVkLWJ5LXRoZS1EYXRhLUNvbXBhcmUtTm9kZS1hbmQtdGhlLWNvcnJlY3BvbnNpbmctdGhlLXVwc2VydC1hbmQtdXBkYXRlLW5vZGVzLTA3LndlYnAiLCJzbGlkZXNob3ciOiI0YmI3NmY2In0%3D" href="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07.webp"><figure class="swiper-slide-inner"><img class="swiper-slide-image swiper-lazy" data-src="https://www.datagaps.com/wp-content/uploads/Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07-1024x576.webp" alt="Different-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-07" /><div class="swiper-lazy-preloader"></div></figure></a></div> </div>
<div class="elementor-swiper-button elementor-swiper-button-prev" role="button" tabindex="0">
<i aria-hidden="true" class="eicon-chevron-left"></i> </div>
<div class="elementor-swiper-button elementor-swiper-button-next" role="button" tabindex="0">
<i aria-hidden="true" class="eicon-chevron-right"></i> </div>
<div class="swiper-pagination"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-ba1c73b elementor-widget elementor-widget-text-editor" data-id="ba1c73b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The different views created by the Data Compare Node and the corresponding upsert and update nodes.</p> </div>
</div>
<div class="elementor-element elementor-element-d9aff85 elementor-widget elementor-widget-text-editor" data-id="d9aff85" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Note that distinguishing between the addition of a new external ID field and updating an existing record is a highly quintessential task especially when the objects are called by CRM and reporting tools themselves where the difference between a new record and an updated record is really critical. </div>
</div>
<div class="elementor-element elementor-element-7b3ef63 elementor-widget elementor-widget-text-editor" data-id="7b3ef63" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The below image shows all views created by the Data Compare Node and the corresponding the upsert and update nodes </div>
</div>
<div class="elementor-element elementor-element-97979c6 elementor-widget elementor-widget-image" data-id="97979c6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="1600" height="900" src="https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01.webp" class="attachment-full size-full wp-image-5970" alt="All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01" srcset="https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01.webp 1600w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-300x169.webp 300w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-1024x576.webp 1024w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-768x432.webp 768w, https://www.datagaps.com/wp-content/uploads/All-views-created-by-the-Data-Compare-Node-and-the-correcponsing-the-upsert-and-update-nodes-01-1536x864.webp 1536w" sizes="(max-width: 1600px) 100vw, 1600px" /> </div>
</div>
<div class="elementor-element elementor-element-5281753 elementor-widget elementor-widget-spacer" data-id="5281753" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-c2e0aa5 elementor-widget elementor-widget-heading" data-id="c2e0aa5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Conclusion</h5> </div>
</div>
<div class="elementor-element elementor-element-8eb153b elementor-widget elementor-widget-text-editor" data-id="8eb153b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In conclusion, it is important to carefully validate Salesforce objects, uploads, and updates to ensure that your data is accurate and consistent. By following best practices and using the appropriate tools and techniques, such as the DataOps Suite, you can avoid common issues such as data formatting errors, data integrity problems, and object-specific issues. Whether you are working with Veeva objects, pre-sales CRM objects, or any other type of object in Salesforce, taking the time to validate your data will help you maintain the quality and reliability of your Salesforce data. </div>
</div>
<div class="elementor-element elementor-element-32e834f elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="32e834f" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-1659015 elementor-widget elementor-widget-heading" data-id="1659015" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Get a Free POC scheduled today!</a></h2> </div>
</div>
<div class="elementor-element elementor-element-d4f2a9e elementor-widget elementor-widget-heading" data-id="d4f2a9e" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">Request Demo</a></h2> </div>
</div>
<div class="elementor-element elementor-element-287ddfd elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="287ddfd" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/validation-of-salesforce-objects-uploads-and-updates/">Validation of Salesforce Objects, Uploads and Updates</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Data Drift Using DataOps Data Profiling</title>
<link>https://www.datagaps.com/blog/data-drift-using-dataops-data-profiling/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Tue, 06 Dec 2022 16:34:31 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7349</guid>
<description><![CDATA[<p>What is Data Drift? Within the data space, the only constant thing is “change”. The drift in data here refers to a multitude of changes in the input data primarily in terms of frequency, aggregates, and heterogeneity. These are not regarded as errors as these types of shifts and changes</p>
<p>The post <a href="https://www.datagaps.com/blog/data-drift-using-dataops-data-profiling/">Data Drift Using DataOps Data Profiling</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7349" class="elementor elementor-7349" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-cb69540 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="cb69540" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-63ee377" data-id="63ee377" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cb5f8c4 elementor-widget elementor-widget-heading" data-id="cb5f8c4" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">What Is Data Drift?</h2> </div>
</div>
<div class="elementor-element elementor-element-a9bd13e elementor-widget elementor-widget-text-editor" data-id="a9bd13e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>What is Data Drift? Within the data space, the only constant thing is “change”. The drift in data here refers to a multitude of changes in the input data primarily in terms of frequency, aggregates, and heterogeneity. These are not regarded as errors as these types of shifts and changes in datasets are factual and representative of the real-world data and how it now differs from the existing benchmarks that a pipeline, analysis, or ML model was initially based on.</p><p>Model Drift comes as the other side of the coin to <a href="https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-monitor-datasets?tabs=python">data drift</a> that is closely related to how the statistical nature and the probabilities as well as the intended logic translation have been altered. While model drift is closely associated with AI-ML models, data drift affects every pipeline that has been made using past production data.</p> </div>
</div>
<div class="elementor-element elementor-element-ec099c6 elementor-widget elementor-widget-heading" data-id="ec099c6" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">A quick way to comprehend data drift is via a couple of real-world examples
</h3> </div>
</div>
<div class="elementor-element elementor-element-3d3cd61 elementor-widget elementor-widget-text-editor" data-id="3d3cd61" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>An ML model that predicts house prices based on a myriad of property aspects such as the number of rooms, area, location, floor, and such which was originally trained in 2019 will not work correctly in 2020 due to the variety of changes in the aforementioned aspects. Certain areas went up in demand as did a certain number of bedrooms and such. If the model is not re-trained or corrected with updated bias, the predicted prices cannot be used.<br />Assume a statistical regression-based model predicts if a customer might default on a loan. The bank’s majority of clients at this point were new families. A few months after the model has been running, the marketing department unveils a new type of campaign targeted toward young students. While the campaign is successful the model is no longer accurate as there are new types of distributions among the various inputs the model is fed. Therefore, the prediction of defaulters itself is incorrect.<br />A reporting system that showcases the mean forecasts across multiple regions suddenly has a higher mean temperature than expected. Under the hood, a few areas had updated their sensors to one of a different brand that resulted in the dimensions being recorded in Fahrenheit as opposed to Celsius on which the system was based.</p><p>A couple of distinctions in the various types of data drifts are the cadence of the drift and the type of the drift. The use cases showed a focus on the type of drift. The cadence of drift segregates drifts into 4 types. These are Sudden Drift, Gradual Drift, Incremental Drift, and Reoccurring Drift. These are usually defined against data distribution and time, but the concept translates with specific aggregates of the metrics themselves.</p> </div>
</div>
<div class="elementor-element elementor-element-9e3e282 elementor-widget elementor-widget-image" data-id="9e3e282" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="435" src="https://www.datagaps.com/wp-content/uploads/Different-Classifications-of-Drift-1024x696.webp" class="attachment-large size-large wp-image-5586" alt="Different-Classifications-of-Drift" srcset="https://www.datagaps.com/wp-content/uploads/Different-Classifications-of-Drift-1024x696.webp 1024w, https://www.datagaps.com/wp-content/uploads/Different-Classifications-of-Drift-300x204.webp 300w, https://www.datagaps.com/wp-content/uploads/Different-Classifications-of-Drift-768x522.webp 768w, https://www.datagaps.com/wp-content/uploads/Different-Classifications-of-Drift-1536x1044.webp 1536w, https://www.datagaps.com/wp-content/uploads/Different-Classifications-of-Drift.webp 1600w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-1ea93b1 elementor-widget elementor-widget-text-editor" data-id="1ea93b1" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 1. Different Classifications of Drift</p> </div>
</div>
<div class="elementor-element elementor-element-a8b5149 elementor-widget elementor-widget-image" data-id="a8b5149" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="324" src="https://www.datagaps.com/wp-content/uploads/Sudden-Drift-1024x518.webp" class="attachment-large size-large wp-image-5590" alt="Sudden-Drift" srcset="https://www.datagaps.com/wp-content/uploads/Sudden-Drift-1024x518.webp 1024w, https://www.datagaps.com/wp-content/uploads/Sudden-Drift-300x152.webp 300w, https://www.datagaps.com/wp-content/uploads/Sudden-Drift-768x388.webp 768w, https://www.datagaps.com/wp-content/uploads/Sudden-Drift.webp 1300w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-b5cf735 elementor-widget elementor-widget-text-editor" data-id="b5cf735" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Figure 2. The above graph showcases a “Sudden” Drift in Yearly Income where the overall values of the metric have increased sharply</p> </div>
</div>
<div class="elementor-element elementor-element-0036f5d elementor-widget elementor-widget-heading" data-id="0036f5d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Profiling as Drift Detection in Data Drift</h2> </div>
</div>
<div class="elementor-element elementor-element-1b72ac1 elementor-widget elementor-widget-text-editor" data-id="1b72ac1" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data Profiling is an integral part of the DataOps Suite that helps users create profiles that hold every aspect of information that can be derived from a dataset such as various types of aggregates such as mean, deviations, min-max, nulls, and more along with Frequency and Pattern Analysis.</p><p>A dataset can be directly pulled into a profiling node. The <a href="https://www.datagaps.com/dataops-suite/dataflow/">DataOps Suite</a> Profile node provides a variety of aggregation and pattern analysis options.</p> </div>
</div>
<div class="elementor-element elementor-element-e5adbd3 elementor-widget elementor-widget-image" data-id="e5adbd3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="522" src="https://www.datagaps.com/wp-content/uploads/DataOps-Profile-Node.webp" class="attachment-large size-large wp-image-5596" alt="DataOps-Profile-Node" srcset="https://www.datagaps.com/wp-content/uploads/DataOps-Profile-Node.webp 956w, https://www.datagaps.com/wp-content/uploads/DataOps-Profile-Node-300x244.webp 300w, https://www.datagaps.com/wp-content/uploads/DataOps-Profile-Node-768x626.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-caf3714 elementor-widget elementor-widget-text-editor" data-id="caf3714" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 3. DataOps Profile Node</p><p>Each of the aggregations works to create a profile of the dataset, maintaining an average value, upper and lower bounds, deviations, patterns, null counts, and such. This help creates a baseline of the expectations in the datasets and something for the users to use for comparisons. Let’s have a closer look at a few real-life examples.</p> </div>
</div>
<div class="elementor-element elementor-element-19764fc elementor-widget elementor-widget-heading" data-id="19764fc" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Data Drift & Variety of Drift Detection – Covariate Drift or Drift in Metrics Stats</h3> </div>
</div>
<div class="elementor-element elementor-element-3dde769 elementor-widget elementor-widget-text-editor" data-id="3dde769" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Every numerical metric holds certain statistical aggregates that can help keep the baseline of the dataset. The most basic ones of these are average, min-max values, and standard deviation. Skewness and Kurtosis also help keep the distribution in check.</p><p>A change in mean implies that in general the average value of the metrics has been altered. In the example below, the yearly income has overall increased.</p> </div>
</div>
<div class="elementor-element elementor-element-9927fc2 elementor-widget elementor-widget-image" data-id="9927fc2" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="331" src="https://www.datagaps.com/wp-content/uploads/Mean.webp" class="attachment-large size-large wp-image-5600" alt="Mean" srcset="https://www.datagaps.com/wp-content/uploads/Mean.webp 877w, https://www.datagaps.com/wp-content/uploads/Mean-300x155.webp 300w, https://www.datagaps.com/wp-content/uploads/Mean-768x398.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-d641c8d elementor-widget elementor-widget-text-editor" data-id="d641c8d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 4. Mean</p><p>While Min-Max values show the upper and lower hard bounds of the metrics, the variability, and the weights away from the mean are showcased by the deviation. In the example we see that while the min and max values of the yearly income have shifted up with the mean, there is less variance in this metric as well, implying that there is less variance in the customers.</p> </div>
</div>
<div class="elementor-element elementor-element-a5e4b43 elementor-widget elementor-widget-image" data-id="a5e4b43" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="245" src="https://www.datagaps.com/wp-content/uploads/Minimum-Value.webp" class="attachment-large size-large wp-image-5604" alt="Minimum-Value" srcset="https://www.datagaps.com/wp-content/uploads/Minimum-Value.webp 876w, https://www.datagaps.com/wp-content/uploads/Minimum-Value-300x115.webp 300w, https://www.datagaps.com/wp-content/uploads/Minimum-Value-768x295.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-047adcd elementor-widget elementor-widget-text-editor" data-id="047adcd" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<div class="et_pb_module et_pb_text et_pb_text_10 et_pb_text_align_left et_pb_bg_layout_light"><div class="et_pb_text_inner"><p style="text-align: center;">Figure 5. Minimum Value</p></div></div> </div>
</div>
<div class="elementor-element elementor-element-a47557c elementor-widget elementor-widget-image" data-id="a47557c" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="174" src="https://www.datagaps.com/wp-content/uploads/Maximum-Value.webp" class="attachment-large size-large wp-image-5608" alt="Maximum-Value" srcset="https://www.datagaps.com/wp-content/uploads/Maximum-Value.webp 884w, https://www.datagaps.com/wp-content/uploads/Maximum-Value-300x81.webp 300w, https://www.datagaps.com/wp-content/uploads/Maximum-Value-768x209.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-a31ad0c elementor-widget elementor-widget-text-editor" data-id="a31ad0c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 6. Maximum Value</p> </div>
</div>
<div class="elementor-element elementor-element-f12fdd4 elementor-widget elementor-widget-image" data-id="f12fdd4" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="276" src="https://www.datagaps.com/wp-content/uploads/Standard-Deviation.webp" class="attachment-large size-large wp-image-5941" alt="Standard-Deviation" srcset="https://www.datagaps.com/wp-content/uploads/Standard-Deviation.webp 883w, https://www.datagaps.com/wp-content/uploads/Standard-Deviation-300x129.webp 300w, https://www.datagaps.com/wp-content/uploads/Standard-Deviation-768x331.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-10cb3c8 elementor-widget elementor-widget-text-editor" data-id="10cb3c8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 7. Standard Deviation [The decrease showcases that most of the values in the past 2 runs are much closer to the mean]</p> </div>
</div>
<div class="elementor-element elementor-element-70edc03 elementor-widget elementor-widget-text-editor" data-id="70edc03" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<span style="text-align: justify; background-color: #ffffff;">Skewness identifies how skewed a dataset is, as in how many values lie evenly away from the mean in both directions while Kurtosis identifies the degree of curve of the distribution of a dataset. Any changes in these datasets represent changes in the distribution and therefore critically affect any statistical tests like the p-test or t-test. In our example, these do not alter as much, however, in more sensitive models such as an AI / ML model, these tiny changes would affect the results more drastically.</span> </div>
</div>
<div class="elementor-element elementor-element-58e15c5 elementor-widget elementor-widget-image" data-id="58e15c5" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="333" src="https://www.datagaps.com/wp-content/uploads/Skeness.webp" class="attachment-large size-large wp-image-5966" alt="Skeness" srcset="https://www.datagaps.com/wp-content/uploads/Skeness.webp 885w, https://www.datagaps.com/wp-content/uploads/Skeness-300x156.webp 300w, https://www.datagaps.com/wp-content/uploads/Skeness-768x399.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-d248c0c elementor-widget elementor-widget-text-editor" data-id="d248c0c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 8. Skewness</p> </div>
</div>
<div class="elementor-element elementor-element-b869cd5 elementor-widget elementor-widget-image" data-id="b869cd5" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="327" src="https://www.datagaps.com/wp-content/uploads/Kurtosis.webp" class="attachment-large size-large wp-image-5969" alt="Kurtosis" srcset="https://www.datagaps.com/wp-content/uploads/Kurtosis.webp 870w, https://www.datagaps.com/wp-content/uploads/Kurtosis-300x153.webp 300w, https://www.datagaps.com/wp-content/uploads/Kurtosis-768x393.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-424b014 elementor-widget elementor-widget-text-editor" data-id="424b014" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 9. Kurtosis</p> </div>
</div>
<div class="elementor-element elementor-element-dbb9887 elementor-widget elementor-widget-heading" data-id="dbb9887" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Change in Keys / GUID</h5> </div>
</div>
<div class="elementor-element elementor-element-c7162d0 elementor-widget elementor-widget-text-editor" data-id="c7162d0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>A GUID or a primary key is on the most important columns in relational datasets. In terms of delta datasets, they are critical in ensuring duplicity doesn’t enter the system. Any changes in these patterns will result in incorrect aggregations and reports, especially when checked against pre-change datasets.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-183afa1 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="183afa1" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-dbd39c6" data-id="dbd39c6" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-fa6b6c7 elementor-widget elementor-widget-image" data-id="fa6b6c7" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="304" height="177" src="https://www.datagaps.com/wp-content/uploads/Before.webp" class="attachment-large size-large wp-image-5971" alt="Before" srcset="https://www.datagaps.com/wp-content/uploads/Before.webp 304w, https://www.datagaps.com/wp-content/uploads/Before-300x175.webp 300w" sizes="(max-width: 304px) 100vw, 304px" /> </div>
</div>
<div class="elementor-element elementor-element-76fafaa elementor-widget elementor-widget-text-editor" data-id="76fafaa" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Before</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-b7189c7" data-id="b7189c7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-fc05c27 elementor-widget elementor-widget-image" data-id="fc05c27" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="314" height="171" src="https://www.datagaps.com/wp-content/uploads/After.webp" class="attachment-large size-large wp-image-5972" alt="After" srcset="https://www.datagaps.com/wp-content/uploads/After.webp 314w, https://www.datagaps.com/wp-content/uploads/After-300x163.webp 300w" sizes="(max-width: 314px) 100vw, 314px" /> </div>
</div>
<div class="elementor-element elementor-element-5e8ae42 elementor-widget elementor-widget-text-editor" data-id="5e8ae42" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">After</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-a4131ff elementor-widget elementor-widget-text-editor" data-id="a4131ff" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the example above, we see that pattern of the Customer Key was a 5-digit number which was suddenly updated to an alphanumeric key.</p> </div>
</div>
<div class="elementor-element elementor-element-f0db4d9 elementor-widget elementor-widget-heading" data-id="f0db4d9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Domain Shift or Addition of New Values</h4> </div>
</div>
<div class="elementor-element elementor-element-8ec355c elementor-widget elementor-widget-text-editor" data-id="8ec355c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>As per the example in the introduction of this blog post if a new campaign type or a new geography id is added to a system the corresponding joins have to be checked. Additionally, if geography id is one of the group columns for any aggregations the aggregates in question are affected as well. While addition that is not in the expected domain is ruled out as a bad record, segregation in teams can result in new validated domain LOVs that the analysis team might not be aware of.</p> </div>
</div>
<div class="elementor-element elementor-element-13699f3 elementor-widget elementor-widget-image" data-id="13699f3" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="640" height="229" src="https://www.datagaps.com/wp-content/uploads/Distinct-Count.webp" class="attachment-large size-large wp-image-5979" alt="Distinct-Count" srcset="https://www.datagaps.com/wp-content/uploads/Distinct-Count.webp 888w, https://www.datagaps.com/wp-content/uploads/Distinct-Count-300x107.webp 300w, https://www.datagaps.com/wp-content/uploads/Distinct-Count-768x275.webp 768w" sizes="(max-width: 640px) 100vw, 640px" /> </div>
</div>
<div class="elementor-element elementor-element-6de9aa8 elementor-widget elementor-widget-text-editor" data-id="6de9aa8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Figure 10. Distinct Count</p> </div>
</div>
<div class="elementor-element elementor-element-276e096 elementor-widget elementor-widget-text-editor" data-id="276e096" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In the example, we see the addition of a few geography ids causing the number of distinct values to vary as well as changes in the distribution of the customers in various geographies.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-216fc3d elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="216fc3d" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-eb9e982" data-id="eb9e982" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-ebb5484 elementor-widget elementor-widget-image" data-id="ebb5484" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="321" height="223" src="https://www.datagaps.com/wp-content/uploads/Before-A.webp" class="attachment-large size-large wp-image-5983" alt="Before-A" srcset="https://www.datagaps.com/wp-content/uploads/Before-A.webp 321w, https://www.datagaps.com/wp-content/uploads/Before-A-300x208.webp 300w" sizes="(max-width: 321px) 100vw, 321px" /> </div>
</div>
<div class="elementor-element elementor-element-ac2ab55 elementor-widget elementor-widget-text-editor" data-id="ac2ab55" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">Before</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-51a9e42" data-id="51a9e42" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-bdae248 elementor-widget elementor-widget-image" data-id="bdae248" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="320" height="232" src="https://www.datagaps.com/wp-content/uploads/After-A.webp" class="attachment-large size-large wp-image-5985" alt="After-A" srcset="https://www.datagaps.com/wp-content/uploads/After-A.webp 320w, https://www.datagaps.com/wp-content/uploads/After-A-300x218.webp 300w" sizes="(max-width: 320px) 100vw, 320px" /> </div>
</div>
<div class="elementor-element elementor-element-0d7163c elementor-widget elementor-widget-text-editor" data-id="0d7163c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p style="text-align: center;">After</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-935f20f elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="935f20f" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
<div class="elementor-element elementor-element-2446033 elementor-widget elementor-widget-heading" data-id="2446033" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Model Drift and Final Thoughts</h5> </div>
</div>
<div class="elementor-element elementor-element-9e84d9c elementor-widget elementor-widget-text-editor" data-id="9e84d9c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Model Drift is the other side of the coin that is affected mainly due to data drift. It refers to degradation in model performance due to changes in data and outdatedness of the model parameters. In a machine learning system only fixing the data drift will not be sufficient and separate techniques will have to be used to detect model drift against production data and model.</p>
<p>Data Drift affects not just ML models but any system that works with functions, aggregates, and systems where statistical tests are being performed. Gradual changes over time creep up in the datasets resulting in lowers data and model quality. If not tracked consistently, figuring out the exact aspects that have changed is a much more complex and difficult task.</p>
<p>Detection of Data Drift is often a de-prioritized task that can be easily deployed, documented, and monitored using the DataOps Profiling Nodes. This will ensure that any type of drift, be it in metrics, domains, patterns or stats can be identified early before the drift causes severe dips in the model quality.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-ab48c48 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="ab48c48" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-wide">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-e62d627" data-id="e62d627" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-a45593d elementor-widget elementor-widget-heading" data-id="a45593d" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">DataOps Suite – Free Trial</h3> </div>
</div>
<div class="elementor-element elementor-element-895da19 elementor-widget elementor-widget-text-editor" data-id="895da19" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>The Datagaps’ DataOps Suite now comes with new components that add extensibility and connectivity with other applications as well as a focus on ease of creating tests by automatically creating SQL Queries and identifying anomalies based on data profile.</p> </div>
</div>
<div class="elementor-element elementor-element-4e834ac elementor-widget elementor-widget-heading" data-id="4e834ac" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">Try DataOps Suite Free for 14 days…</h3> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0ac7d59" data-id="0ac7d59" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-931e2ba elementor-button-align-stretch elementor-widget elementor-widget-form" data-id="931e2ba" data-element_type="widget" data-e-type="widget" data-settings="{"step_next_label":"Next","step_previous_label":"Previous","button_width":"100","step_type":"number_text","step_icon_shape":"circle"}" data-widget_type="form.default">
<div class="elementor-widget-container">
<form class="elementor-form" method="post" name="New Form" aria-label="New Form">
<input type="hidden" name="post_id" value="7349"/>
<input type="hidden" name="form_id" value="931e2ba"/>
<input type="hidden" name="referer_title" value="What is Data Drift Detection | Data Drift in DataOps Suite" />
<input type="hidden" name="queried_id" value="7349"/>
<div class="elementor-form-fields-wrapper elementor-labels-above">
<div class="elementor-field-type-text elementor-field-group elementor-column elementor-field-group-name elementor-col-100">
<input size="1" type="text" name="form_fields[name]" id="form-field-name" class="elementor-field elementor-size-sm elementor-field-textual" placeholder="Full name">
</div>
<div class="elementor-field-type-email elementor-field-group elementor-column elementor-field-group-email elementor-col-100 elementor-field-required">
<input size="1" type="email" name="form_fields[email]" id="form-field-email" class="elementor-field elementor-size-sm elementor-field-textual" placeholder="Email" required="required">
</div>
<div class="elementor-field-type-html elementor-field-group elementor-column elementor-field-group-field_05ac09e elementor-col-100">
<p style="color:black;font-size:12px;font-weight:bold;margin-top:10px;">Please check the box below</p> </div>
<div class="elementor-field-type-checkbox elementor-field-group elementor-column elementor-field-group-field_762c09a elementor-col-100">
<div class="elementor-field-subgroup "><span class="elementor-field-option"><input type="checkbox" value="I agree Datagaps and/or its representatives to reach me for further communication about the course details and any relevant Datagaps information." id="form-field-field_762c09a-0" name="form_fields[field_762c09a]"> <label for="form-field-field_762c09a-0">I agree Datagaps and/or its representatives to reach me for further communication about the course details and any relevant Datagaps information.</label></span></div> </div>
<div class="elementor-field-group elementor-column elementor-field-type-submit elementor-col-100 e-form__buttons">
<button class="elementor-button elementor-size-sm" type="submit">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Request Free Trial</span>
</span>
</button>
</div>
</div>
</form>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-bd45442 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="bd45442" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/data-drift-using-dataops-data-profiling/">Data Drift Using DataOps Data Profiling</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>6 Critical Components of Data Testing</title>
<link>https://www.datagaps.com/blog/6-critical-components-of-data-testing/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Mon, 05 Dec 2022 12:41:17 +0000</pubDate>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[DevOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7381</guid>
<description><![CDATA[<p>Data is a precious asset that has to be validated at various stages of use. One stage is at the point of ingestion, and another as it moves through your enterprise and lands in your data warehouse or data lake. Finally, when it is consumed in your data analytics platform.</p>
<p>The post <a href="https://www.datagaps.com/blog/6-critical-components-of-data-testing/">6 Critical Components of Data Testing</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7381" class="elementor elementor-7381" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-338839b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="338839b" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-77b6450" data-id="77b6450" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0d68b17 elementor-widget elementor-widget-heading" data-id="0d68b17" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Importance of Data and Data Testing</h2> </div>
</div>
<div class="elementor-element elementor-element-7d2423a elementor-widget elementor-widget-text-editor" data-id="7d2423a" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Data is a precious asset that has to be validated at various stages of use. One stage is at the point of ingestion, and another as it moves through your enterprise and lands in your data warehouse or data lake. Finally, when it is consumed in your data analytics platform. This is from the point of view of analyzing data.</p><p>What about all of the production data that you have in the enterprise?</p><p>How is that going to be monitored?</p><p>So, table stakes for <a href="https://en.wikipedia.org/wiki/Database_testing">data testing</a> start with access to all the data in your environment, whether in your analytics platform or stored within your production applications. Along with the data access, data quality rules have to be available, as well as a method of comparing data sources of like or mixed data structures and varying volumes, often in the billions.</p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-8463ead elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="8463ead" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-047834a" data-id="047834a" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-c56d628 elementor-widget elementor-widget-text-editor" data-id="c56d628" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<ul><li>With these core capabilities, you can develop good testing workflows that take care of <strong>75%</strong> of your testing needs.</li><li>But what about the other <strong>25%</strong>?</li><li>What if your data is in complex hierarchical JSON structures?</li><li>What if the data testing needs are not anticipated and solved?</li></ul><p>The last 25% brings about the 6 critical components where you can solve those unexpected needs.</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-f670666" data-id="f670666" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-19ebe85 elementor-widget elementor-widget-image" data-id="19ebe85" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/data.webp" class="attachment-large size-large wp-image-5591" alt="data" srcset="https://www.datagaps.com/wp-content/uploads/data.webp 512w, https://www.datagaps.com/wp-content/uploads/data-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/data-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-2652777 elementor-widget elementor-widget-heading" data-id="2652777" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h2 class="elementor-heading-title elementor-size-default">Here are the 6 critical components</h2> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-7abcc25 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="7abcc25" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-bd9506c" data-id="bd9506c" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cf82061 elementor-widget elementor-widget-text-editor" data-id="cf82061" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><span style="color: #ffffff;">Extensibility</span></p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0e382cf" data-id="0e382cf" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-f4eaf66 elementor-widget elementor-widget-text-editor" data-id="f4eaf66" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In data testing, there are often times when you need to be able to extend your solution to other areas that weren’t anticipated. A unique data problem is encountered that is outside the norm and could not be thought of beforehand. For example, ADD A COUPLE EXAMPLES. If your solution is extensible through Python or some other method, the issue can be resolved quickly. <a href="https://www.youtube.com/watch?v=hZnVo7nZGpg">With Datagaps, we provide a Plugin component that can be selected from a library of components that is extensible by using Python.</a> This eliminates the need for complex workarounds that you have to shoehorn into other solutions.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-944d71e elementor-widget elementor-widget-spacer" data-id="944d71e" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-9d5f443 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="9d5f443" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-956285f" data-id="956285f" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d18e39b elementor-widget elementor-widget-text-editor" data-id="d18e39b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<span style="color: #ffffff;">Advanced API Components</span> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-7eccac7" data-id="7eccac7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-727947b elementor-widget elementor-widget-text-editor" data-id="727947b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In today’s world, data comes to us in a variety of ways. Often as simple as CSV files, feeds from production applications or data that is FTP’d to a location. Quite often, there are requirements to use an Advanced API to get access to the data. In one recent example our client had 8 API’s that we needed to invoke to gain access to their Hierarchical JSON data. We needed to create multiple files from each of the API’s which meant that we needed advanced capabilities. That is the point of our API component which easily handled the clients needs.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-5089869 elementor-widget elementor-widget-spacer" data-id="5089869" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-17d2c40 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="17d2c40" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-61284e2" data-id="61284e2" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-ecbb133 elementor-widget elementor-widget-text-editor" data-id="ecbb133" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<span style="color: #ffffff;">AI Based Observabilitys</span> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-3155a5b" data-id="3155a5b" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8557900 elementor-widget elementor-widget-text-editor" data-id="8557900" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Writing Data Quality rules is effective in most situations, but often it may not be needed if your solution can learn from the data being ingested. A combination of Data Quality rules and Data Observability is the best approach. Data Quality rules can surface likely data issues efficiently while Data Observability will find outliers that haven’t been anticipated before. You may try Datagaps <a href="https://www.datagaps.com/dataops-suite/data-quality/">Data Quality Monitor</a> for this.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-c72a11b elementor-widget elementor-widget-spacer" data-id="c72a11b" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-57613d3 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="57613d3" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-e1587ff" data-id="e1587ff" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-554cebc elementor-widget elementor-widget-text-editor" data-id="554cebc" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<span style="color: #ffffff;">Ability To Handle Large Volumes in the Billions</span> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-f0bc608" data-id="f0bc608" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-5457ebe elementor-widget elementor-widget-text-editor" data-id="5457ebe" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>As data volumes continue to grow through a variety of means, at some point in time your normal processing requirements will grow to challenge your data testing capabilities. We provide two means to do data comparisons. The first is through a database engine that handles up to 40 million rows and is easier to set up and cost a little less to do the data comparisons. The second method that covers high volumes is apache spark based in memory comparisons. This method takes advantage of native cloud capabilities such as clusters and auto scaling. So if you volumes are low currently the DB Engine will take care of the volumes but as your data scales you have an option to swap out the DB Engine for the Apache spark implementation that can meet your current of future needs. <a href="https://www.datagaps.com/blog/how-do-you-automate-big-data-testing-everything-to-know/">Learn more about automating your Big Data.</a></p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-f85fb02 elementor-widget elementor-widget-spacer" data-id="f85fb02" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-4f38410 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="4f38410" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-a1cdfbf" data-id="a1cdfbf" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8a84cac elementor-widget elementor-widget-text-editor" data-id="8a84cac" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<span style="color: #ffffff;">Integration with your DevOps Platform</span> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-db604de" data-id="db604de" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-e388336 elementor-widget elementor-widget-text-editor" data-id="e388336" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Your DevOps organization has spent an enormous amount of time and cost to implement a DevOps platform. As you introduce your DataOps platform it is important to be able to integrate with the DevOps platform such as x,y,z. This ensures consistency between how your DevOps ad DataOps process execution and management.</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-6ceacf0 elementor-widget elementor-widget-spacer" data-id="6ceacf0" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-0d7ebac elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0d7ebac" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-8fd2f01" data-id="8fd2f01" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-ef3dcad elementor-widget elementor-widget-text-editor" data-id="ef3dcad" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<span style="color: #ffffff;">Integration with an RPA Platform</span> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-f3e1f92" data-id="f3e1f92" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cc5b1ba elementor-widget elementor-widget-text-editor" data-id="cc5b1ba" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Python, Scala and SQL use cases can be extended to handle a limitless number of variations in your data test plans. However, these languages, while easy to use for developers aren’t meant for the business user. Additionally, they aren’t designed to mimic human behavior. There is a Billion dollar industry that caters to Robotic Process Automation. In other words, RPA mimics the human interaction</p> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-b742638 elementor-widget elementor-widget-spacer" data-id="b742638" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<div class="elementor-element elementor-element-18f4b83 elementor-widget elementor-widget-text-editor" data-id="18f4b83" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>In conclusion, data testing needs have risen in importance as organizations monetize the use of the data or make critical decisions based on the data flowing through their enterprise. Volumes are increasing, sources take on different access methods, and often, data needs to be accessed through alternative means via API or other methods. Your processing needs have certainly grown substantially in the past few years. Methods of testing are changing rapidly. That is why we believe extensibility is so important. As all of these dynamics impact your business and future needs, a platform that will scale and extend capabilities will be critical for current and future needs.</p> </div>
</div>
<div class="elementor-element elementor-element-1b6c07b elementor-widget elementor-widget-heading" data-id="1b6c07b" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">
Get a Free POC scheduled today!</a></h3> </div>
</div>
<div class="elementor-element elementor-element-d2324b0 elementor-widget elementor-widget-heading" data-id="d2324b0" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default"><a href="https://www.datagaps.com/request-demo/">
Request Demo</a></h3> </div>
</div>
<div class="elementor-element elementor-element-8234874 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="8234874" data-element_type="widget" data-e-type="widget" data-widget_type="divider.default">
<div class="elementor-widget-container">
<div class="elementor-divider">
<span class="elementor-divider-separator">
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/6-critical-components-of-data-testing/">6 Critical Components of Data Testing</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
<item>
<title>DataOps Suite Feature Updates – Version 2022.5.0</title>
<link>https://www.datagaps.com/blog/dataops-suite-feature-updates-version-2022-5-0/</link>
<dc:creator><![CDATA[Rajesh Kumar]]></dc:creator>
<pubDate>Thu, 03 Nov 2022 13:07:37 +0000</pubDate>
<category><![CDATA[BI Testing]]></category>
<category><![CDATA[Cloud Data Migration]]></category>
<category><![CDATA[Dataflow]]></category>
<category><![CDATA[DataOps]]></category>
<category><![CDATA[ETL Testing]]></category>
<guid isPermaLink="false">https://staging9.datagaps.com/?p=7238</guid>
<description><![CDATA[<p>Datagaps Inc. announced its latest product feature updates for the DataOps Suite Version 2022.5.0 – The theme of this update is Integration and Easy to apply Analysis. If you are interested in getting a demo of DataOps Suite, you can fill your details here.</p>
<p>The post <a href="https://www.datagaps.com/blog/dataops-suite-feature-updates-version-2022-5-0/">DataOps Suite Feature Updates – Version 2022.5.0</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></description>
<content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="7238" class="elementor elementor-7238" data-elementor-post-type="post">
<section class="elementor-section elementor-top-section elementor-element elementor-element-fed3cc0 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="fed3cc0" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-50026b7" data-id="50026b7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-f042f70 elementor-widget elementor-widget-heading" data-id="f042f70" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default"><a title="Datagaps" href="https://www.datagaps.com/" target="_blank" rel="noopener"><b>Datagaps Inc.</b></a> announced its latest product feature updates for the DataOps Suite Version 2022.5.0 – The theme of this update is Integration and Easy to apply Analysis. If you are interested in getting a demo of DataOps Suite, you can fill your details here.</h5> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-a96efab elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="a96efab" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4c08e4f" data-id="4c08e4f" data-element_type="column" data-e-type="column" data-settings="{"background_background":"classic"}">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8579bec elementor-widget elementor-widget-heading" data-id="8579bec" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">DataOps Suite Top Feature Updates – Version 2022.5.0</h3> </div>
</div>
</div>
</div>
</div>
</section>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-9180a93 elementor-section-content-middle bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="9180a93" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-16 elementor-inner-column elementor-element elementor-element-d64e56d" data-id="d64e56d" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-96e6a9a elementor-widget elementor-widget-image" data-id="96e6a9a" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="150" height="150" src="https://www.datagaps.com/wp-content/uploads/API-150x150.webp" class="attachment-thumbnail size-thumbnail wp-image-6547" alt="API" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-16 elementor-inner-column elementor-element elementor-element-5bbf0e3" data-id="5bbf0e3" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-26890fd elementor-widget elementor-widget-image" data-id="26890fd" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="150" height="150" src="https://www.datagaps.com/wp-content/uploads/Plugin-150x150.webp" class="attachment-thumbnail size-thumbnail wp-image-6551" alt="Plugin" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-16 elementor-inner-column elementor-element elementor-element-b996dd8" data-id="b996dd8" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-509dd59 elementor-widget elementor-widget-image" data-id="509dd59" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="150" height="150" src="https://www.datagaps.com/wp-content/uploads/Metric-Compare-150x150.webp" class="attachment-thumbnail size-thumbnail wp-image-6553" alt="Metric-Compare" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-16 elementor-inner-column elementor-element elementor-element-45878d5" data-id="45878d5" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-78538cd elementor-widget elementor-widget-image" data-id="78538cd" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="150" height="150" src="https://www.datagaps.com/wp-content/uploads/Data-Profile-1-150x150.webp" class="attachment-thumbnail size-thumbnail wp-image-6555" alt="Data-Profile" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-16 elementor-inner-column elementor-element elementor-element-16949c5" data-id="16949c5" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-cadb134 elementor-widget elementor-widget-image" data-id="cadb134" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="150" height="150" src="https://www.datagaps.com/wp-content/uploads/Data-Analysis-150x150.webp" class="attachment-thumbnail size-thumbnail wp-image-6556" alt="Data-Analysis" /> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-16 elementor-inner-column elementor-element elementor-element-d350361" data-id="d350361" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-98c6250 elementor-widget elementor-widget-image" data-id="98c6250" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
<div class="elementor-widget-container">
<img loading="lazy" decoding="async" width="150" height="150" src="https://www.datagaps.com/wp-content/uploads/SQL-Query-Builder-150x150.webp" class="attachment-thumbnail size-thumbnail wp-image-6558" alt="SQL-Query-Builder" /> </div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-4d91d68 elementor-widget elementor-widget-spacer" data-id="4d91d68" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-f1bee8e elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="f1bee8e" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-636ece4" data-id="636ece4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0236ff2 elementor-widget elementor-widget-heading" data-id="0236ff2" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default"><a href="#api-component">API Component</a></h5> </div>
</div>
<div class="elementor-element elementor-element-af214f9 elementor-widget elementor-widget-text-editor" data-id="af214f9" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>REST APIs are being used extensively for Application development as well as a means of integration between systems. DataOps suite now supports API as a data source and the corresponding API Component in DataFlow can be used to call the REST API and automatically convert the response into datasets for further validation and processing.</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ac8ad60" data-id="ac8ad60" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-50cae4a elementor-widget elementor-widget-video" data-id="50cae4a" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/www.youtube.com\/embed\/7BiqdeH5IBQ?feature=oembed","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-919dc0d elementor-widget elementor-widget-text-editor" data-id="919dc0d" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Since the REST API responses can be deeply hierarchical JSON documents, DataOps suite automatically splits the data into separate datasets with relations so that anyone with SQL query knowledge can easily query the REST API output and validate the data by comparing it with data from another data source. </div>
</div>
<div class="elementor-element elementor-element-135173c elementor-widget elementor-widget-spacer" data-id="135173c" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-14dd7d2 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="14dd7d2" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-d288eb4" data-id="d288eb4" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-b84c74f elementor-widget elementor-widget-heading" data-id="b84c74f" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default"><a href="#Plugin-Component">Plugin Component</a></h5> </div>
</div>
<div class="elementor-element elementor-element-4e3e6a4 elementor-widget elementor-widget-text-editor" data-id="4e3e6a4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p>Plugin component adds extensibility to DataOps suite. The plugin system lets users define methods with user-defined parameters as inputs which once defined, can be called on with a few clicks by the end user. Plugins can be written in Python or Scala. Once configured, they can be used within the DataFlow without the need for copy-pasting the code thus promoting code re-use.</p> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ef554e3" data-id="ef554e3" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0ed70b5 elementor-widget elementor-widget-video" data-id="0ed70b5" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/www.youtube.com\/embed\/n6RgHeKFd1s?feature=oembed","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-9d2791c elementor-widget elementor-widget-text-editor" data-id="9d2791c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
One of the example plugin we recently created was to read multiple sheets of excel (usually reports received by mail) as separate datasets to be used for comparisons. </div>
</div>
<div class="elementor-element elementor-element-128c64e elementor-widget elementor-widget-spacer" data-id="128c64e" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-b7a5aff elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="b7a5aff" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-4a3bf91" data-id="4a3bf91" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-ec3f039 elementor-widget elementor-widget-heading" data-id="ec3f039" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Metrics Compare Component</h5> </div>
</div>
<div class="elementor-element elementor-element-b77c562 elementor-widget elementor-widget-text-editor" data-id="b77c562" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Metrics component makes it easy to compare metrics such as count, sum etc. across different systems. When it comes to production data monitoring, it is a common requirement to compare the counts of source and target records to validate that the ETL job processes all the records as expected. </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-e0d2cb7" data-id="e0d2cb7" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-83de0da elementor-widget elementor-widget-video" data-id="83de0da" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/www.youtube.com\/embed\/sfBdXepEog4?feature=oembed","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-b161946 elementor-widget elementor-widget-text-editor" data-id="b161946" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Another feature of Metrics component is that it can be used to compare multiple metrics in a single component. These metrics can be executed in parallel for better performance. </div>
</div>
<div class="elementor-element elementor-element-0bcadfc elementor-widget elementor-widget-spacer" data-id="0bcadfc" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-23e8972 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="23e8972" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-7b93f87" data-id="7b93f87" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0f331ea elementor-widget elementor-widget-heading" data-id="0f331ea" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Data Profile Component</h5> </div>
</div>
<div class="elementor-element elementor-element-7eb4541 elementor-widget elementor-widget-text-editor" data-id="7eb4541" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Data Profile component has been available in DataOps suite for a very long time. It takes a dataset as an input and automatically computes aggregates such as maximum value, minimum value, minimum length, maximum length, Null (%), distinct (%), distinct count, Null count, mean, sum, row count, Constancy, Kurtosis, Skewness, and standard deviation. </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-76b4340" data-id="76b4340" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-ce946d8 elementor-widget elementor-widget-video" data-id="ce946d8" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/www.youtube.com\/embed\/gtvrUdQnUkw?feature=oembed","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-4facd5b elementor-widget elementor-widget-text-editor" data-id="4facd5b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
In this release, Data Profile component can be used to identify anomalies in data automatically based on the historical data profile values. For example, if an ETL job is ingesting 1000 records on daily basis and if one day it only processed 100 records, Data Profile component will automatically flag that as an anomaly. </div>
</div>
<div class="elementor-element elementor-element-1ac77a7 elementor-widget elementor-widget-text-editor" data-id="1ac77a7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
This observability feature of Data Profile component can be used to automatically detect anomalies in other data profile values such as % of Nulls in a column or Mean value. </div>
</div>
<div class="elementor-element elementor-element-b962de6 elementor-widget elementor-widget-spacer" data-id="b962de6" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-3857c58 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="3857c58" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-787965d" data-id="787965d" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-8b1acb9 elementor-widget elementor-widget-heading" data-id="8b1acb9" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">Data Observability Improvements</h5> </div>
</div>
<div class="elementor-element elementor-element-ae5018b elementor-widget elementor-widget-text-editor" data-id="ae5018b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Data Analysis component automatically learns from historical data and identifies anomalies in the latest data. Within the realms of data validation, anomaly detection plays one of the most crucial roles as it is not defined with strict logic or rules but with patterns and aggregates. Some of the prediction methods supported include time series based machine learning models as well statistical methods such as IQR and Standard Deviation. </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-7299aea" data-id="7299aea" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-844216b elementor-widget elementor-widget-video" data-id="844216b" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/www.youtube.com\/embed\/ZcZSNjifyRc?feature=oembed","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-af65397 elementor-widget elementor-widget-text-editor" data-id="af65397" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
Version 2022.5.0 simplifies anomaly detection by providing the option to do inline prediction. The results screen was also improved to show anomalies that were identified during the run. </div>
</div>
<div class="elementor-element elementor-element-e795b17 elementor-widget elementor-widget-spacer" data-id="e795b17" data-element_type="widget" data-e-type="widget" data-widget_type="spacer.default">
<div class="elementor-widget-container">
<div class="elementor-spacer">
<div class="elementor-spacer-inner"></div>
</div>
</div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-5b7cea6 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="5b7cea6" data-element_type="section" data-e-type="section">
<div class="elementor-container elementor-column-gap-default">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-f71559c" data-id="f71559c" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-9f53226 elementor-widget elementor-widget-heading" data-id="9f53226" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h5 class="elementor-heading-title elementor-size-default">SQL Query Builder</h5> </div>
</div>
<div class="elementor-element elementor-element-93f60ff elementor-widget elementor-widget-text-editor" data-id="93f60ff" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The SQL Query Builder option simplifies the generation of SQL queries from the data model without writing the query manually. It has three sections: Entities, Columns, and Conditions.
<ul>
<li><strong>The Entities</strong> section displays the tables and their columns corresponding to the schema of the selected data model.</li>
</ul> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-246a145" data-id="246a145" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-0864657 elementor-widget elementor-widget-video" data-id="0864657" data-element_type="widget" data-e-type="widget" data-settings="{"youtube_url":"https:\/\/www.youtube.com\/embed\/EERc3QQIAxw?feature=oembed","video_type":"youtube","controls":"yes"}" data-widget_type="video.default">
<div class="elementor-widget-container">
<div class="elementor-wrapper elementor-open-inline">
<div class="elementor-video"></div> </div>
</div>
</div>
</div>
</div>
</div>
</section>
<div class="elementor-element elementor-element-5f44092 elementor-widget elementor-widget-text-editor" data-id="5f44092" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<ul>
<li><strong>The Columns</strong> section displays the columns that will be used to generate the SQL query with or without the usage of the aggregate function.</li>
<li><strong>The Conditions</strong> section allows the user to generate the WHERE clause condition as per the end user’s requirement.</li>
</ul>
In addition to above-mentioned updates, a lot of additions and enhancements were made to Data Rules, Test Cases, Test Data Manger (TDM) and CLI. Version 2022.5.1 also adds support for Azure AD SSO integration. </div>
</div>
<div class="elementor-element elementor-element-2f3f47e elementor-widget elementor-widget-text-editor" data-id="2f3f47e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
<p><a href="https://help.datagaps.com/articles/#!dataops-suite/dataops-suite-releases">Full Release Notes of DataOps Suite – Learn More</a></p> </div>
</div>
<section class="elementor-section elementor-inner-section elementor-element elementor-element-29f1738 elementor-section-content-top bw-ac elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="29f1738" data-element_type="section" data-e-type="section" data-settings="{"background_background":"classic"}">
<div class="elementor-container elementor-column-gap-wide">
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-64b7728" data-id="64b7728" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-d000e53 elementor-widget elementor-widget-heading" data-id="d000e53" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h3 class="elementor-heading-title elementor-size-default">DataOps Suite – Free Trial</h3> </div>
</div>
<div class="elementor-element elementor-element-ee08a0b elementor-widget elementor-widget-text-editor" data-id="ee08a0b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
<div class="elementor-widget-container">
The Datagaps’ DataOps Suite now comes with new components that add extensibility and connectivity with other applications as well as a focus on ease of creating tests by automatically creating SQL Queries and identifying anomalies based on data profile. </div>
</div>
<div class="elementor-element elementor-element-901f8d5 elementor-widget elementor-widget-heading" data-id="901f8d5" data-element_type="widget" data-e-type="widget" data-widget_type="heading.default">
<div class="elementor-widget-container">
<h4 class="elementor-heading-title elementor-size-default">Try DataOps Suite Free for 14 days…</h4> </div>
</div>
</div>
</div>
<div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-51e4685" data-id="51e4685" data-element_type="column" data-e-type="column">
<div class="elementor-widget-wrap elementor-element-populated">
<div class="elementor-element elementor-element-45cf5f0 elementor-button-align-stretch elementor-widget elementor-widget-form" data-id="45cf5f0" data-element_type="widget" data-e-type="widget" data-settings="{"step_next_label":"Next","step_previous_label":"Previous","button_width":"100","step_type":"number_text","step_icon_shape":"circle"}" data-widget_type="form.default">
<div class="elementor-widget-container">
<form class="elementor-form" method="post" name="New Form" aria-label="New Form">
<input type="hidden" name="post_id" value="7238"/>
<input type="hidden" name="form_id" value="45cf5f0"/>
<input type="hidden" name="referer_title" value="DataOps Suite Feature Updates – Version 2022.5.0" />
<input type="hidden" name="queried_id" value="7238"/>
<div class="elementor-form-fields-wrapper elementor-labels-above">
<div class="elementor-field-type-text elementor-field-group elementor-column elementor-field-group-name elementor-col-100">
<input size="1" type="text" name="form_fields[name]" id="form-field-name" class="elementor-field elementor-size-sm elementor-field-textual" placeholder="Full name">
</div>
<div class="elementor-field-type-email elementor-field-group elementor-column elementor-field-group-email elementor-col-100 elementor-field-required">
<input size="1" type="email" name="form_fields[email]" id="form-field-email" class="elementor-field elementor-size-sm elementor-field-textual" placeholder="Email" required="required">
</div>
<div class="elementor-field-type-html elementor-field-group elementor-column elementor-field-group-field_05ac09e elementor-col-100">
<p style="color:#444444;font-size:10px;margin-bottom:0;letter-spacing:1px;font-weight:600;line-height:20px;">Please check the box below</p> </div>
<div class="elementor-field-type-checkbox elementor-field-group elementor-column elementor-field-group-field_762c09a elementor-col-100">
<div class="elementor-field-subgroup "><span class="elementor-field-option"><input type="checkbox" value="I agree Datagaps and/or its representatives to reach me for further communication about the course details and any relevant Datagaps information." id="form-field-field_762c09a-0" name="form_fields[field_762c09a]"> <label for="form-field-field_762c09a-0">I agree Datagaps and/or its representatives to reach me for further communication about the course details and any relevant Datagaps information.</label></span></div> </div>
<div class="elementor-field-group elementor-column elementor-field-type-submit elementor-col-100 e-form__buttons">
<button class="elementor-button elementor-size-sm" type="submit">
<span class="elementor-button-content-wrapper">
<span class="elementor-button-text">Request Free Trial</span>
</span>
</button>
</div>
</div>
</form>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
</div>
</div>
</section>
</div>
<p>The post <a href="https://www.datagaps.com/blog/dataops-suite-feature-updates-version-2022-5-0/">DataOps Suite Feature Updates – Version 2022.5.0</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Automated Cloud Data Testing | ETL, BI & BigData</a>.</p>
]]></content:encoded>
</item>
</channel>
</rss>