A Data Quality measures how well a dataset meets criteria for accuracy, completeness, validity, consistency, uniqueness, timeliness and fitness for purpose, and it is critical to all data governance initiatives within an organization. (topic source from IBM)
What is Data Quality Scorecard?
How do you know that the data quality is good? Data engineers and analysts require a proactive approach to maintaining high-quality data pipelines. Datagaps DataOps Suite comes with a Data Quality Scorecard mechanism. This score is calculated on the basis of user-defined rules to perform data quality checks.
As data is processed, the scorecard checks each record against these rules. Passing rules increases the score, while failing ones decreases it, giving teams a transparent and quantifiable measure of data quality. This offers a real-time, data-driven metric for assessing quality.
DataOps Suite’s Data Quality Monitor can help users perform rule checks of data to make sure the data is right, irrespective of whether it is a model or table or a record. It also provides an overall data quality scorecard template which is an aggregated score of all the data models present in the application.
Overall Aggregate Data Quality Score
Data Quality Score for Data Model
Similarly, we can have table wise data score as well where quality of the data is scored by column depending on the rules associated with them.
And the following screenshot describes how data quality rules help in scoring the quality of the data. It is a result of a sample rule run.
Data Observability through Datagaps DataOps suite
Data observability refers to the practice of monitoring, managing and maintaining data in a way that ensures its quality, availability and reliability across various processes, systems and pipelines within an organization. (What is data observability? – Source of topic from IBM)
With Datagaps DataOps Suite, organizations can achieve real-time Data Observability by proactively identifying data anomalies, structural changes, and missing records, helping businesses maintain clean and reliable data.
The “Data Observability” component in DataOps Suite is a user-friendly component for Statistical calculations (STD, IQR, Time Series, Fixed Deviation, and Delta Deviation) to report data anomalies.
This identifies one-off anomalies that skew the anomaly calculations and ignores them. This is achieved with the help of Machine Learning Algorithms.
This component can perform AI-driven predictions and detect the anomalies of incoming or existing data using Machine Learning.
So, if there is any irregular high in the data, the application catches the differences in the pattern of graphs.
- The Standard Deviation statistical method detects the variation of data based on the mean and variance. If any observation is beyond the upper or lower bound value, then it is an anomaly or outlier.
- The Inter Quartile Range or IQR (Q3 – Q1) is another statistical method to detect anomalies by dividing the dataset into quartiles. Low outliers are determined when the 1.5*IQR is below the first quartile (Q1 – 1.5*IQR). High outliers are determined when the 1.5*IQR is above the third quartile (Q3 + 1.5*IQR).
- Time Series is a collection of quantities that are assembled over even intervals in time and ordered chronologically. The time interval at which data is collected is generally referred to as the time series frequency.
- Fixed Deviation is an anomaly detection method where the upper and lower bound values are user-defined and fixed. Any data point deviating from the expected upper and lower threshold values will be considered anomalies or outliers. The lower threshold value can also range from negative (e.g., -100).
- Delta Deviation is an anomaly detection method where the upper and lower threshold values vary based on the input value specified in the upper and lower variance respectively. The upper and lower variances are user-defined in percentages. Any data point deviating from the expected upper and lower threshold values will be considered anomalies or outliers.
After selecting the source dataset, users are taken to the columns section where users can choose the appropriate columns from the dataset columns. They can group them together if required. This will help in categorizing the columns for predicting/analyzing the target data.
Similarly, the appropriate columns can be chosen in “Measures” section on which the anomaly detection is to be performed. Aggregates such as MIN, MAX, SUM and others can be applied to these columns.
Next in the observability component comes the most important part, where users are prompted to choose the type of prediction method. You can see the prediction section for the IQR prediction method below.
If we observe the screenshot, we can find some mandatory fields filled. These mandatory fields are the necessary parameters for that specific prediction method to calculate and detect the anomalies.
IQR constant is an empirical value which can be changed based on the distribution of data.
Minimum data point is the minimum number of data points taken into consideration to perform the statistical calculations for accurate predictions.
The Rolling Window is used in the statistical calculation to determine the upper and lower bound values of the current data based on the number of past values.
Data Quality Rule Examples: If the Rolling Window is 8, the lower and upper bound values of the current data will be predicted based on the previous values (8 days value).
The “should not consider negative values” checkbox ignores the negative lower bound value and is replaced with “Zero“.
The Incremental Run checkbox is enabled to perform the data analysis of the latest data that is added to the source table daily.
Similarly, we have other important terminologies, like lower and upper variance, Seasonality, Confidence interval, No. of Future Predictions, which is the value that is used to predict the number of future days’ lower and upper bounds.
So, the component gives enough flexibility for users to consider various parameters and fine-tune them as required because the needs, goals, processes and the data itself varies from organization to organization.
After running the prediction, the result would look like this
And on clicking fail, the resulting graph would look like this
Empowering Data Quality Through Observability
Data Observability component leverages machine learning which helps the application to learn expected patterns in the data and flags anomalies when the data deviates from these learned boundaries. This approach complements the data quality checks as these two can be combined to create a robust framework for maintaining high-quality data across an organization’s pipelines.
Data Observability isn’t just about spotting outliers, it drives continuous improvement and serves as a powerful catalyst for enhancing the effectiveness of existing data quality rules.
It acts as a proactive layer over rule-based monitoring, ensuring continuous improvement in data quality. With regular evaluation of incoming data, Data observability complements rule-based monitoring by detecting anomalies that static checks might miss. Even when Data Quality Scores remain high according to existing rules, observability can uncover hidden issues that lead to incorrect insights.
By leveraging observability, users can identify these issues and refine their rules proactively, ensuring that their monitoring framework remains proactive and responsive.
As the data landscape evolves, so must our approach to managing it. By combining rule-based monitoring with observability, organizations can stay ahead of potential issues and ensure that their data remains accurate and reliable. With DataGaps DataOps Suite, you gain the tools to adapt, ensuring every decision is powered by high–quality data.
Enhance Your Data Quality with DataOps Suite
Establish trust through continuous data scoring and advanced Data Observability to maintain high-quality data pipelines. Take control with real-time insights and anomaly detection today.
