<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" > <channel> <title>David Small, Author at Datagaps | Gen AI-Powered Automated Cloud Data Testing</title> <atom:link href="https://www.datagaps.com/blog/author/david_small/feed/" rel="self" type="application/rss+xml" /> <link></link> <description></description> <lastBuildDate>Thu, 18 Jan 2024 13:28:23 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod> <sy:updateFrequency> 1 </sy:updateFrequency> <generator>https://wordpress.org/?v=6.9.4</generator> <image> <url>https://www.datagaps.com/wp-content/uploads/Datagaps-India-Favicon-Lite-theme-150x150.jpg</url> <title>David Small, Author at Datagaps | Gen AI-Powered Automated Cloud Data Testing</title> <link></link> <width>32</width> <height>32</height> </image> <item> <title>5 Critical Steps To Achieve Trustworthy Data in your Data Warehouse</title> <link>https://www.datagaps.com/blog/5-critical-steps-to-achieve-trustworthy-data-in-your-data-warehouse/</link> <dc:creator><![CDATA[David Small]]></dc:creator> <pubDate>Tue, 17 Jan 2023 10:07:05 +0000</pubDate> <category><![CDATA[BI Testing]]></category> <category><![CDATA[Cloud Data Migration]]></category> <category><![CDATA[Data Quality]]></category> <category><![CDATA[Data Validation]]></category> <category><![CDATA[DataOps]]></category> <category><![CDATA[ETL Testing]]></category> <guid isPermaLink="false">https://staging9.datagaps.com/?p=9958</guid> <description><![CDATA[<p>Data Warehouses provide a comprehensive view of data across many different sources, and proper analysis can encourage better business decisions and problem-solving. However, building and maintaining an effective data warehouse requires careful thought and consideration</p> <p>The post <a href="https://www.datagaps.com/blog/5-critical-steps-to-achieve-trustworthy-data-in-your-data-warehouse/">5 Critical Steps To Achieve Trustworthy Data in your Data Warehouse</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p> ]]></description> <content:encoded><![CDATA[ <div data-elementor-type="wp-post" data-elementor-id="9958" class="elementor elementor-9958" data-elementor-post-type="post"> <section class="elementor-section elementor-top-section elementor-element elementor-element-c266c1a elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="c266c1a" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-no"> <div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f89d2c9" data-id="f89d2c9" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-0bfbaf7 elementor-widget elementor-widget-text-editor" data-id="0bfbaf7" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><a href="https://aws.amazon.com/data-warehouse/">Data Warehouses</a> provide a comprehensive view of data across many different sources, and proper analysis can encourage better business decisions and problem-solving. However, building and maintaining an effective data warehouse requires careful thought and consideration to ensure that it is reliable and can withstand the business demands of the organizations that rely on it. The biggest concern for data warehouse users is whether they can trust the information it produces.</p> </div> </div> <div class="elementor-element elementor-element-a64546f elementor-widget elementor-widget-text-editor" data-id="a64546f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>They can use the data to make informed decisions and implement successful business strategies if the data is trustworthy. But if the data quality is terrible, stakeholders risk making bad decisions that lead to unnecessary expenses, wasted effort, staff management issues, and many other negative impacts that directly and indirectly hurt the business.</p> </div> </div> <section class="elementor-section elementor-inner-section elementor-element elementor-element-ebfc1ea elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="ebfc1ea" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-968226c" data-id="968226c" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-05ec210 elementor-widget elementor-widget-image" data-id="05ec210" data-element_type="widget" data-e-type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img fetchpriority="high" decoding="async" width="640" height="335" src="https://www.datagaps.com/wp-content/uploads/Accurate-01.webp" class="attachment-large size-large wp-image-9960" alt="Accurate-01" srcset="https://www.datagaps.com/wp-content/uploads/Accurate-01.webp 690w, https://www.datagaps.com/wp-content/uploads/Accurate-01-300x157.webp 300w" sizes="(max-width: 640px) 100vw, 640px" /> </div> </div> </div> </div> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-784d531" data-id="784d531" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-d60042f elementor-widget elementor-widget-text-editor" data-id="d60042f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>Knowing the rewards and risks involved with data accuracy, the development team must devote a large part of their time to building effective data warehouse testing methods. Many organizations do not follow a consistent or comprehensive approach to testing the quality of their data warehouses.</p> </div> </div> </div> </div> </div> </section> <div class="elementor-element elementor-element-81ab9a0 elementor-widget elementor-widget-text-editor" data-id="81ab9a0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>They are often overwhelmed by the complexity of managing multiple data sources and all the potential issues that could arise throughout the development process. By taking a systematic approach to testing a data warehouse, developers can continually address challenges as they arise and improve the quality of their data.</p> </div> </div> <div class="elementor-element elementor-element-3dbed41 elementor-widget elementor-widget-text-editor" data-id="3dbed41" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>Also Read: <a href="/blog/top-5-must-haves-in-your-data-testing-platform/">Top 5 Must Haves In Your Data Testing Platform</a></p> </div> </div> <div class="elementor-element elementor-element-671c788 elementor-widget elementor-widget-text-editor" data-id="671c788" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>Here are five critical tests that should be used during development to ensure high-quality data in data warehouses.</p> </div> </div> <section class="elementor-section elementor-inner-section elementor-element elementor-element-689d3f5 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="689d3f5" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-ba8a7ee" data-id="ba8a7ee" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-7da8015 elementor-widget elementor-widget-image" data-id="7da8015" data-element_type="widget" data-e-type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/number-one.webp" class="attachment-large size-large wp-image-9961" alt="number-one" srcset="https://www.datagaps.com/wp-content/uploads/number-one.webp 512w, https://www.datagaps.com/wp-content/uploads/number-one-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/number-one-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div> </div> </div> </div> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-a3fadac" data-id="a3fadac" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-4bfc252 elementor-widget elementor-widget-text-editor" data-id="4bfc252" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><strong>Ensure that ETL processes run reliably by continuously monitoring the data flowing into the warehouse.</strong> This type of testing is essential for large projects with many moving parts and complex dependencies between the different datasets. For example, suppose you are working with a data warehouse incorporating data from multiple source systems, perhaps storing similar data but with varying transformation steps. If you analyze your dataset without monitoring the results throughout your development cycle, you may find that output values are incorrect or might be missing altogether. As a general rule, the longer you avoid monitoring the data for accuracy, the harder it becomes to trace back to the root source of the data and identify any fixes. To overcome this, tools that continuously monitor the source data during extraction, transformation, and loading (ETL) will ensure that the data is appropriately captured, allowing you to quickly identify anomalies in the results that require further investigation.</p> </div> </div> </div> </div> </div> </section> <section class="elementor-section elementor-inner-section elementor-element elementor-element-348d635 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="348d635" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0643243" data-id="0643243" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-43a7dee elementor-widget elementor-widget-image" data-id="43a7dee" data-element_type="widget" data-e-type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/2-1.webp" class="attachment-large size-large wp-image-9962" alt="2" srcset="https://www.datagaps.com/wp-content/uploads/2-1.webp 512w, https://www.datagaps.com/wp-content/uploads/2-1-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/2-1-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div> </div> </div> </div> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-daaf553" data-id="daaf553" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-59d87b8 elementor-widget elementor-widget-text-editor" data-id="59d87b8" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><strong>Make sure that the output of your transformations is accurate by running validation procedures after each transformation step.</strong> For example, if you modify your inbound data multiple times from the source to the target, you should validate the expected data after each transformation. By ensuring these validations, you will be more focused on the small details you might otherwise overlook. This test will allow you to identify any problems more efficiently and to ensure that they are corrected before delivering the new data to the warehouse.</p> </div> </div> </div> </div> </div> </section> <section class="elementor-section elementor-inner-section elementor-element elementor-element-d8b0956 elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="d8b0956" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-257d60d" data-id="257d60d" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-04e675f elementor-widget elementor-widget-image" data-id="04e675f" data-element_type="widget" data-e-type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/number-3.webp" class="attachment-large size-large wp-image-9963" alt="three" srcset="https://www.datagaps.com/wp-content/uploads/number-3.webp 512w, https://www.datagaps.com/wp-content/uploads/number-3-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/number-3-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div> </div> </div> </div> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-0c56c0e" data-id="0c56c0e" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-a46fc97 elementor-widget elementor-widget-text-editor" data-id="a46fc97" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><strong>Execute cleaning and scrubbing procedures against your incorrect data.</strong> Many source systems, especially those in use for an extended period, can contain a lot of noise in the form of incomplete or inaccurate data. For example, a CRM system might have customer records with old addresses that are no longer valid, while other records might include outdated or incomplete contact names. Although ETL test cases assume that data remains unchanged from source to target, ‘dirty’ data can skew the results of your analysis and lead to unreliable results and inflated statistics. By building tests that identify the most common inaccuracies and transform them into the correct values, you will ensure that your warehouse incorporates higher-quality data into its operation.</p> </div> </div> </div> </div> </div> </section> <section class="elementor-section elementor-inner-section elementor-element elementor-element-d9c249a elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="d9c249a" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-d3efdc7" data-id="d3efdc7" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-c53b4d9 elementor-widget elementor-widget-image" data-id="c53b4d9" data-element_type="widget" data-e-type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/4-1.webp" class="attachment-large size-large wp-image-9964" alt="four" srcset="https://www.datagaps.com/wp-content/uploads/4-1.webp 512w, https://www.datagaps.com/wp-content/uploads/4-1-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/4-1-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div> </div> </div> </div> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-01def82" data-id="01def82" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-0da4879 elementor-widget elementor-widget-text-editor" data-id="0da4879" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><strong>Review your source system processes and configurations regularly to identify potential impacts caused by database schema changes.</strong> For example, business logic changes might require source systems to track new fields in your data warehouse, such as marketing codes or business segments. If you miss the downstream updates, you might inadvertently include invalid data in your warehouse. In addition, the relationships identified between your tables often change as new data is added and removed., causing records to be overwritten or corrupted. Regular reviews will ensure your source systems capture the correct schema modifications before they affect the quality of your data.</p> </div> </div> </div> </div> </div> </section> <section class="elementor-section elementor-inner-section elementor-element elementor-element-c91534f elementor-section-content-top bw-ac elementor-section-full_width elementor-section-height-default elementor-section-height-default" data-id="c91534f" data-element_type="section" data-e-type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-5d95cb6" data-id="5d95cb6" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-cd949cb elementor-widget elementor-widget-image" data-id="cd949cb" data-element_type="widget" data-e-type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img loading="lazy" decoding="async" width="512" height="512" src="https://www.datagaps.com/wp-content/uploads/number-5.webp" class="attachment-large size-large wp-image-9965" alt="number-5" srcset="https://www.datagaps.com/wp-content/uploads/number-5.webp 512w, https://www.datagaps.com/wp-content/uploads/number-5-300x300.webp 300w, https://www.datagaps.com/wp-content/uploads/number-5-150x150.webp 150w" sizes="(max-width: 512px) 100vw, 512px" /> </div> </div> </div> </div> <div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-8190662" data-id="8190662" data-element_type="column" data-e-type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-4e1bec0 elementor-widget elementor-widget-text-editor" data-id="4e1bec0" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><strong>Verify the rules of your data warehouse regularly by adding unique Test Cases across the various source systems.</strong> While you ideally want to map accurate data from start to finish, including specific exception logic to ‘fix’ insufficient data, you should also include deletable source test data to verify the accuracy of your transformation steps and identify any errors that might occur in the data on its journey into the warehouse. For example, you might want to add a duplicate record on a field that might never show duplicate records to see how the system performs, or you might deliberately create an undefined lookup value to demonstrate how this error is handled. These results will help you confirm that the warehouse can process the output from the source systems under all conditions and also will provide an accurate representation of the enterprise data when the time comes.</p> </div> </div> </div> </div> </div> </section> <div class="elementor-element elementor-element-aa9c3f4 elementor-widget elementor-widget-text-editor" data-id="aa9c3f4" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>Although these five test recommendations may seem straightforward, designing them can be complicated and time-consuming without the benefit of a specific starting point. At the very minimum, begin with a flexible testing approach that focuses on validating the accuracy of the data delivered to the warehouse for one <a href="https://www.datagaps.com/data-testing-concepts/etl-testing/">ETL testing</a> use case. Then add more complexity to your scripts, incorporating the other suggestions. Remember that your ultimate goal is to create a solid test plan that produces reliable data at its core and will deliver reliable and meaningful results for your stakeholders. How you reach this goal does not matter as long as you use proven, practical approaches.</p> </div> </div> <div class="elementor-element elementor-element-efdcfce elementor-widget elementor-widget-text-editor" data-id="efdcfce" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p>Fortunately, there are several tools that you can use to accelerate your test development efforts. Whether you are building data warehouse tests from scratch or building out legacy scripts that have been in use for years, DataGaps and its suite of high-performance testing tools can help you identify data quality issues and build a robust testing framework that you can replicate and reuse throughout the project lifecycle. For example, with the DataOps Suite, you can streamline the ETL testing and quality assurance process while empowering your team to easily create and manage thousands of test cases on the fly.</p> </div> </div> <div class="elementor-element elementor-element-86f0998 elementor-widget elementor-widget-text-editor" data-id="86f0998" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><a href="https://www.datagaps.com/">Datagaps</a> DataOps suite has a library of ready-to-use adaptors to popular data sources, such as Talend, Informatica, SAP, and Oracle that makes it possible to automate complex test data workflows without writing a single line of code, providing an intuitive interface that allows you to build, test, and deploy pipelines with relative ease. You can generate complex data test scripts and documents quickly, guaranteeing your users receive the trusted information they need to make better and more informed business decisions.</p><h3><a href="https://www.datagaps.com/request-demo/">To know more, request a demo now!</a></h3> </div> </div> </div> </div> </div> </section> </div> <p>The post <a href="https://www.datagaps.com/blog/5-critical-steps-to-achieve-trustworthy-data-in-your-data-warehouse/">5 Critical Steps To Achieve Trustworthy Data in your Data Warehouse</a> appeared first on <a href="https://www.datagaps.com">Datagaps | Gen AI-Powered Automated Cloud Data Testing</a>.</p> ]]></content:encoded> </item> </channel> </rss>