Data Quality Scorecards measure dataset quality using user-defined rules, giving teams a real-time, quantifiable score at the model, table, or column level. This blog also covers Data Observability, which uses statistical methods—Standard Deviation, IQR, Time Series, Fixed Deviation, and Delta Deviation—powered by machine learning to detect anomalies rule-based checks might miss. Together, scorecards and observability form a feedback loop: observability uncovers hidden issues even when scores look healthy, helping teams continuously refine and strengthen their data quality rules.
Key Takeaways
- Data Quality Scorecards quantify quality in real time — each record is checked against user-defined rules, with passing/failing rules raising or lowering the score at the model, table, or column level. datagaps
- Five statistical methods power anomaly detection — Standard Deviation, IQR, Time Series, Fixed Deviation, and Delta Deviation each detect anomalies differently, using either fixed thresholds, quartile ranges, or user-defined variance percentages.
- Machine learning filters out one-off noise — the Data Observability component uses ML algorithms to ignore isolated anomalies that would otherwise skew calculations, improving detection accuracy over time.
- Observability complements — not replaces — rule-based checks — even when Data Quality Scores appear healthy under existing rules, observability can surface hidden issues, creating a continuous feedback loop that helps refine rules over time.
A Data Quality measures how well a dataset meets criteria for accuracy, completeness, validity, consistency, uniqueness, timeliness and fitness for purpose, and it is critical to all data governance initiatives within an organization. (topic source from IBM)
What is Data Quality Scorecard?
How do you know that the data quality is good? Data engineers and analysts require a proactive approach to maintaining high-quality data pipelines. Datagaps DataOps Suite comes with a Data Quality Scorecard mechanism. This score is calculated on the basis of user-defined rules to perform data quality checks.
As data is processed, the scorecard checks each record against these rules. Passing rules increases the score, while failing ones decreases it, giving teams a transparent and quantifiable measure of data quality. This offers a real-time, data-driven metric for assessing quality.
DataOps Suite’s Data Quality Monitor can help users perform rule checks of data to make sure the data is right, irrespective of whether it is a model or table or a record. It also provides an overall data quality scorecard template which is an aggregated score of all the data models present in the application.
Overall Aggregate Data Quality Score
Data Quality Score for Data Model
Similarly, we can have table wise data score as well where quality of the data is scored by column depending on the rules associated with them.
And the following screenshot describes how data quality rules help in scoring the quality of the data. It is a result of a sample rule run.
Data Observability through Datagaps DataOps suite
Data observability refers to the practice of monitoring, managing and maintaining data in a way that ensures its quality, availability and reliability across various processes, systems and pipelines within an organization. (What is data observability? – Source of topic from IBM)
With Datagaps DataOps Suite, organizations can achieve real-time Data Observability by proactively identifying data anomalies, structural changes, and missing records, helping businesses maintain clean and reliable data.
The “Data Observability” component in DataOps Suite is a user-friendly component for Statistical calculations (STD, IQR, Time Series, Fixed Deviation, and Delta Deviation) to report data anomalies.
This identifies one-off anomalies that skew the anomaly calculations and ignores them. This is achieved with the help of Machine Learning Algorithms.
This component can perform AI-driven predictions and detect the anomalies of incoming or existing data using Machine Learning.
So, if there is any irregular high in the data, the application catches the differences in the pattern of graphs.
- The Standard Deviation statistical method detects the variation of data based on the mean and variance. If any observation is beyond the upper or lower bound value, then it is an anomaly or outlier.
- The Inter Quartile Range or IQR (Q3 – Q1) is another statistical method to detect anomalies by dividing the dataset into quartiles. Low outliers are determined when the 1.5*IQR is below the first quartile (Q1 – 1.5*IQR). High outliers are determined when the 1.5*IQR is above the third quartile (Q3 + 1.5*IQR).
- Time Series is a collection of quantities that are assembled over even intervals in time and ordered chronologically. The time interval at which data is collected is generally referred to as the time series frequency.
- Fixed Deviation is an anomaly detection method where the upper and lower bound values are user-defined and fixed. Any data point deviating from the expected upper and lower threshold values will be considered anomalies or outliers. The lower threshold value can also range from negative (e.g., -100).
- Delta Deviation is an anomaly detection method where the upper and lower threshold values vary based on the input value specified in the upper and lower variance respectively. The upper and lower variances are user-defined in percentages. Any data point deviating from the expected upper and lower threshold values will be considered anomalies or outliers.
After selecting the source dataset, users are taken to the columns section where users can choose the appropriate columns from the dataset columns. They can group them together if required. This will help in categorizing the columns for predicting/analyzing the target data.
Similarly, the appropriate columns can be chosen in “Measures” section on which the anomaly detection is to be performed. Aggregates such as MIN, MAX, SUM and others can be applied to these columns.
Next in the observability component comes the most important part, where users are prompted to choose the type of prediction method. You can see the prediction section for the IQR prediction method below.
If we observe the screenshot, we can find some mandatory fields filled. These mandatory fields are the necessary parameters for that specific prediction method to calculate and detect the anomalies.
IQR constant is an empirical value which can be changed based on the distribution of data.
Minimum data point is the minimum number of data points taken into consideration to perform the statistical calculations for accurate predictions.
The Rolling Window is used in the statistical calculation to determine the upper and lower bound values of the current data based on the number of past values.
Data Quality Rule Examples: If the Rolling Window is 8, the lower and upper bound values of the current data will be predicted based on the previous values (8 days value).
The “should not consider negative values” checkbox ignores the negative lower bound value and is replaced with “Zero“.
The Incremental Run checkbox is enabled to perform the data analysis of the latest data that is added to the source table daily.
Similarly, we have other important terminologies, like lower and upper variance, Seasonality, Confidence interval, No. of Future Predictions, which is the value that is used to predict the number of future days’ lower and upper bounds.
So, the component gives enough flexibility for users to consider various parameters and fine-tune them as required because the needs, goals, processes and the data itself varies from organization to organization.
After running the prediction, the result would look like this
And on clicking fail, the resulting graph would look like this
Empowering Data Quality Through Observability
Data Observability component leverages machine learning which helps the application to learn expected patterns in the data and flags anomalies when the data deviates from these learned boundaries. This approach complements the data quality checks as these two can be combined to create a robust framework for maintaining high-quality data across an organization’s pipelines.
Data Observability isn’t just about spotting outliers, it drives continuous improvement and serves as a powerful catalyst for enhancing the effectiveness of existing data quality rules.
It acts as a proactive layer over rule-based monitoring, ensuring continuous improvement in data quality. With regular evaluation of incoming data, Data observability complements rule-based monitoring by detecting anomalies that static checks might miss. Even when Data Quality Scores remain high according to existing rules, observability can uncover hidden issues that lead to incorrect insights.
By leveraging observability, users can identify these issues and refine their rules proactively, ensuring that their monitoring framework remains proactive and responsive.
As the data landscape evolves, so must our approach to managing it. By combining rule-based monitoring with observability, organizations can stay ahead of potential issues and ensure that their data remains accurate and reliable. With DataGaps DataOps Suite, you gain the tools to adapt, ensuring every decision is powered by high–quality data.
FAQs: Data Quality Scorecards and Data Observability
1) What is a Data Quality Scorecard?
A Data Quality Scorecard provides a measurable view of data quality by evaluating datasets against user-defined validation rules. It generates real-time quality scores at the model, table, or column level based on the percentage of records that pass or fail those rules.
2) What statistical methods are used for data observability and anomaly detection?
Data Observability supports multiple statistical approaches, including Standard Deviation, Interquartile Range (IQR), Time Series, Fixed Deviation, and Delta Deviation. Each method detects anomalies using different techniques, such as historical trends, quartile analysis, fixed thresholds, or user-defined variance limits.
3) How does machine learning improve anomaly detection accuracy?
Machine learning enhances anomaly detection by distinguishing genuine recurring patterns from isolated or one-time outliers. This reduces false positives and enables Data Observability to focus on meaningful data quality issues rather than statistical noise.
4) How do Data Quality Scorecards and Data Observability work together?
Data Quality Scorecards measure compliance with predefined quality rules, while Data Observability identifies unexpected data behavior that existing rules may not detect. Together, they create a continuous improvement cycle where observability insights help refine validation rules and strengthen overall data quality.
Enhance Your Data Quality with DataOps Suite
Establish trust through continuous data scoring and advanced Data Observability to maintain high-quality data pipelines. Take control with real-time insights and anomaly detection today.
