Data observability is the continuous monitoring of data health across the full lifecycle, from ingestion to reporting. The Datagaps DataOps Suite delivers it on Microsoft Fabric with AI-generated rules, data quality scoring, ETL and BI validation, profiling and ML-based anomaly detection across Bronze, Silver and Gold layers.
Key Takeaways
- A green pipeline and a green dashboard can still hide wrong data, such as overnight duplicate records.
- Data quality checks data at rest; data observability tracks data in motion across the full lifecycle.
- The DataOps Suite maps onto Fabric’s medallion architecture across Bronze, Silver and Gold layers on OneLake.
- Generated rules remain the same more than 80% of the time, so they can be exported and reused.
- AI spans column descriptions, rule generation, PII classification, natural-language SQL and anomaly detection.
End-to-end data observability on Microsoft Fabric means monitoring data health continuously as it moves from ingestion through transformation, storage and reporting — not just running data quality checks on tables. Datagaps delivers this with the DataOps Suite, which validates and monitors data at multiple levels and integrates with existing architectures.
Why do green pipelines still produce untrustworthy data?
Imagine waking up in the morning and seeing revenue on your dashboard jump by 40%. Everyone celebrates until someone realises that a pipeline duplicated records overnight. The pipeline was green. The dashboard was green. Everything looked fine, except the data itself was wrong.
Over the last five years, the importance of data has grown exponentially. Organisations have started using platforms like Microsoft Fabric and Databricks to build solutions, but the trust factor remains missing. The reason is straightforward: data pipelines cannot be fully trusted until they are fully tested. Modern solutions require not just AI development, but also end-to-end AI testing.
Beyond trust, data observability also addresses accountability. Without clear visibility into where data went wrong and why, issues become difficult to trace and responsibility becomes hard to assign across engineering, analytics, and business teams.
What is data observability?
Data observability is the monitoring of the entire data lifecycle, from ingestion to reporting. It is a concept that is frequently misunderstood, and its meaning often varies depending on who is working with the data and how they interpret it. To understand it properly, it helps to look at it through the lens of different roles.
What does data observability mean for a data engineer?
As a data engineer, the primary concern is separating good data from bad data. If a data quality check fails, you want to know exactly where the issue lies, whether it is in the source, a transformation, or a downstream table. That visibility is what observability means at this level.
What does data observability mean for an analytics manager?
At a management level, observability goes further. It is not just about finding data quality issues; it is about linking failures to data flow lineage to understand root causes, monitoring data quality health over time, prioritising fixes that require more attention, and having a measurable view of the entire pipeline to report to stakeholders.
What does data observability mean for a data scientist?
For a data scientist, observability means performing more aggressive testing of the data before it reaches models or dashboards. This includes anomaly detection using ML techniques, such as identifying an unusual shift in data volume or a sudden increase in values within a column beyond its expected range. These signals serve as early warnings, giving the team the opportunity to investigate and resolve issues before potentially corrupted or drifted data is fed into production models.
How is data observability different from data quality?
Data quality focuses on whether data meets predefined standards at a point in time, while data observability tracks data health continuously as it moves through systems, pipelines, and reports. The two terms are often used interchangeably, but they address different concerns. The comparison below clarifies the distinction:
| Dimension | Data Observability | Data Quality |
|---|---|---|
| Scope | Data in motion; full lifecycle from ingestion to reporting | Data at rest; focuses on tables and datasets |
| Issue Detection | Detects known and unknown issues proactively | Checks known issues; mostly reactive |
| Coverage | Drift, anomalies, lineage, and ML-based monitoring | Nulls, duplicates, and static rule validation |
| Approach | Proactive with automated alerting | Static, predefined rules |
| Primary Use | End-to-end pipeline health and traceability | Table-level data integrity checks |
Data quality focuses on whether data meets predefined standards at a point in time. Data observability takes a broader view, tracking data health continuously as it moves through systems, pipelines, and reports.
How does Datagaps enable end-to-end data observability on Microsoft Fabric?
Datagaps offers the DataOps Suite to address the trust problem described above. It is an end-to-end testing and monitoring platform that integrates with existing architectures and performs validation and monitoring at multiple levels. The components of the DataOps Suite work across the full analytics pipeline, regardless of the platform.
For Microsoft Fabric environments specifically, the integration maps natively onto the medallion architecture, with Datagaps components operating across Bronze, Silver, and Gold layers on OneLake.
How do AI-generated column descriptions set up the data model?
The process begins as soon as you connect your data source and create a data model. The first task is generating descriptions for your columns using AI. This produces consistent metadata that the platform uses to understand your dataset and generate better rules and queries wherever required.
![Datagaps column description generation using AI on DimCustomer table with Generate Descriptions button highlighted]](https://www.datagaps.com/wp-content/uploads/Datagaps-column-description.3.png)
Once descriptions are generated and sample data has been reviewed, the next step is building data quality processes, starting with defining data quality rules.
How does Datagaps generate data quality rules automatically?
Datagaps generates data quality rules automatically for all selected tables, without requiring manual checks to be written. The AI uses sample data and metadata to produce rules covering dimensions such as:
- Validity: ensuring values conform to expected formats and constraints
- Uniqueness: detecting duplicate records or primary key violations
- Completeness: identifying null or missing values
- Consistency: validating conformity of related fields
- Timeliness: checking that data arrives within expected time windows
A practical advantage of this approach is reusability. Once rules are generated for a given set of tables, more than 80% of the time those rules remain the same across runs. They can be exported, imported, and customised rather than regenerated each time, which also reduces AI processing costs.
How does Datagaps detect and classify PII automatically?
For datasets containing personally identifiable information, Datagaps uses machine learning to automatically detect and classify PII columns across all tables. Rather than manually tagging sensitive fields, the platform classifies each column into a known entity type such as email address, phone number, or national identifier. These columns can then be masked or flagged for governance purposes without any additional code.
How is the data quality score calculated?
Every rule defined in a table contributes to an individual data quality score at the column level. These roll up into an aggregated score visible in the data quality dashboard. This gives teams a standardised, quantifiable view of data health that can be tracked over time and broken down by dimension.
All records that fail defined quality checks are surfaced in the Data Quality Issues section.
How are data quality issues managed and remediated?
From the Data Quality Issues interface, teams can choose to save, quarantine, or delete bad records. Notifications, stakeholder alerts, and ticket creation can all be automated from within the platform, ensuring full accountability throughout the remediation process. For example, if a check is added to ensure that first and last names in a table contain no special characters or numbers, the platform will show exactly where that check failed, how many bad records were found, and the sample rows that triggered the failure. The team can then decide whether to retain, isolate, or remove those records before proceeding.
How do dataflows validate ETL transformations and compare data?
A key part of end-to-end observability is verifying that transformations produce the expected output. The best way to find mismatches is by creating separate dataflows for testing. A dataflow connects a raw source, applies a transformation, and then runs a comparison to surface any rows where the output does not match expectations.
The result view shows exactly which rows were mismatched, the original value versus the changed value per column, and a column-wise breakdown of discrepancies. This level of clarity is essential before moving any data solution to production, and is particularly valuable during data migration scenarios where column mappings and type conversions need to be validated.
Within Dataflows, an AI Assistant is also available to help write correct and complex SQL queries using simple natural language prompts. To give the AI accurate context, a RAG model of the data schema can be configured, allowing the assistant to generate queries that are grounded in the actual metadata of the dataset.
How are BI reports validated against source data?
Observability does not end at the database layer. Reports also need to be validated against the original dataset. As an analytics manager, checking data quality scores alone is insufficient; the figures shown in a report must also match what is in the source tables. Dataflows support this by allowing a comparison between a BI tool dataset and the underlying source.
For example, if a report on Tableau is aggregating yearly income by number of children, the same aggregation can be written against the source table and compared directly. The tool picks up the dataset from the dashboard automatically and validates whether the numbers match. Any discrepancy in values is surfaced as a mismatched row, including type mismatches such as a column appearing as a decimal in the source but as a string in the report.
How do data profiling and anomaly detection work?
A minimal yet important capability that also falls under data observability is profiling your data and identifying patterns. For datasets that change every day, profiling provides a report on what has changed, what the distribution looks like, and where anomalies may have appeared.
Beyond standard aggregates such as minimum value, maximum value, null count, and mean, the platform supports machine learning-based analysis to detect skewness and hidden patterns in datasets. This does not require any custom code; selecting the desired profiling options is sufficient and the platform generates the report automatically.
The output is a detailed profile showing observed patterns per column, frequency of values, and trend comparisons against previous runs, giving teams a continuous view of how their data is behaving over time.
What role does AI play across the Datagaps observability framework?
In the DataOps Suite, AI is embedded across the entire observability workflow rather than sitting in one feature. AI used for development in Microsoft Fabric is now matched by AI for testing as well. The AI capabilities span the following areas:
- AI-generated column descriptions to establish consistent and machine-readable metadata
- AI-powered rule generation based on sample data and metadata context
- ML-based entity classification for automated PII detection across all tables
- An AI Assistant within Dataflows for generating SQL queries using natural language prompts, grounded in a RAG model of the data schema
- ML-based anomaly detection for identifying distribution shifts and statistical irregularities without requiring custom code
Conclusion
The best thing about having a data observability setup in place is that you can compare what goes into the system against what is being reflected on the dashboard or in your tables. You can introduce tests at every stage to validate what is happening and whether it is correct.
As organisations scale their data operations on Microsoft Fabric, data quality checks alone are not sufficient. Observability needs continuous visibility across the full data lifecycle, from ingestion through transformation, storage, and reporting. Datagaps and its DataOps Suite provide the tooling to achieve this, with AI accelerating every stage of the process and integrating natively with existing architectures.
Frequently Asked Questions
What is the difference between data observability and data quality?
Data quality checks whether data at rest meets predefined standards, using static rules for issues like nulls and duplicates. Data observability covers data in motion across the full lifecycle, proactively detecting known and unknown issues through drift, anomaly, lineage and ML-based monitoring with automated alerting. Data quality checks are one part of observability.
How does Datagaps support data observability on Microsoft Fabric?
The Datagaps DataOps Suite integrates with existing architectures and maps natively onto Fabric’s medallion architecture, with Data Quality Monitor, ETL Validation and BI Validation operating across Bronze, Silver and Gold layers on OneLake. It covers AI-generated rules, DQ scoring, issue management, dataflow-based comparisons, BI report validation and profiling.
Can AI-generated data quality rules be reused across runs?
Yes. Once rules are generated for a given set of tables, more than 80% of the time those rules remain the same across runs. They can be exported, imported and customised rather than regenerated each time, which also reduces AI processing costs. Rules cover validity, uniqueness, completeness, consistency and timeliness.
What happens to bad records that fail data quality checks in Datagaps?
Failed records are surfaced in the Data Quality Issues section, where teams can save, quarantine or delete them. The platform shows where a check failed, how many bad records were found and sample rows. Notifications, stakeholder alerts and ticket creation can be automated, ensuring accountability throughout remediation.
How do you validate that a BI report matches its source data?
Datagaps dataflows compare a BI tool dataset with the underlying source. For a Tableau report aggregating yearly income by number of children, the same aggregation is written against the source table and compared directly. Discrepancies appear as mismatched rows, including type mismatches such as a decimal in the source shown as a string in the report.
RajMohan Achanta
Associate Product Manager, Datagaps
Associate Product Manager at Datagaps. Shapes the product experience across ETL Validator, BI Validator, and Data Quality Monitor.
Avinash Keshri
Head, Product Marketing
Head of Product Marketing at Datagaps and IIM Bangalore alumnus. 13+ years of experience in commercializing AI and data platforms across global markets.
