Data Profiling In Pharma Datasets Using DataOps Suite

Data profiling is a crucial step in the data management process, especially in the pharmaceutical industry where accurate and reliable data is essential for making informed decisions. Data profiling involves examining and summarizing the characteristics of a dataset in order to identify patterns, trends, and anomalies in the data. By tracking aggregations and patterns in the data, it is possible to identify potential issues or anomalies that may need to be addressed in order to improve the quality of the data.

Pattern Recognition and Tracking of Keys and Strings

In the pharmaceutical industry, it is common for different vendors to provide datasets that contain information on the same subjects or entities. For example, a vendor may provide a dataset containing information on clinical trial participants, while another vendor may provide a dataset containing information on patient outcomes.

In order to accurately merge or join these datasets, it is important that the primary keys used to identify the subjects or entities are consistent. For example, if one dataset uses a 9-digit numerical key to identify participants, it is important that any other datasets that contain information on the same participants also use a 9-digit numerical key.

If the pattern of the primary keys is not consistent, it can make it difficult or impossible to accurately link records from different datasets. This can lead to errors or incorrect analyses and can compromise the overall integrity of the data.

As seen in the example below, originally the only pattern seen in the datasets was a 9-digit key. However, in the latest run post, an update from the client we see a new alphanumeric pattern is also seen in the system. This might indicate a data-type change and a definite notification in data governance.

DataOps Suite: Data profile node result showcasing a change in the patterns of a primary key

Outliers in Patient Claims and Drug Sales Datasets

Outliers are values in a dataset that are significantly different from the majority of the other values. In inpatient claims and drug sales datasets, outliers can occur in various aggregates, such as averages, standard deviations, minimum values, and maximum values.

Outliers can have a significant impact on the results of any analyses or modeling efforts, as they can distort the overall patterns or trends in the data. For example, if a dataset contains an outlier value that is significantly higher or lower than the majority of the other values, it could skew the average or standard deviation, leading to incorrect or misleading results.

Try DataOps Suite – Free Trial

A few examples of how variations in min-max values and standard deviations can help identify anomalies in patient claims and drug sales datasets:

• • • If the minimum value for a dataset decreases significantly over time, it could indicate an anomaly or error in the data. For example, if the minimum value for a column containing drug prices decreases significantly from one month to the next, it could indicate that the price was entered incorrectly or that the drug is being sold at a significantly discounted rate.
    • If the maximum value for a dataset increases significantly over time, it could also indicate an anomaly or error in the data. For example, if the maximum value for a column containing drug prices increases significantly from one month to the next, it could indicate that the price was entered incorrectly or that the drug is being sold at a significantly inflated rate.
    • If the standard deviation for a dataset increases significantly over time, it could also indicate an anomaly or error in the data. For example, if the standard deviation for a column containing drug prices increases significantly from one month to the next, it could indicate that the prices are becoming more variable than expected, which could be a sign of an anomaly or error.

DataOps Suite: Data Profile Node Results

Also Read: Data Drift Using DataOps Data Profiling

Distributions and List of Values Deltas

For inpatient claims and drug sales datasets, it is important to monitor the distribution of values across different columns and variables. The DataOps Suite’s profile node can provide various plots and statistics that can help you understand the distribution of values in your data.

For example, if you are analyzing a dataset containing information on patient claims, you might be interested in the distribution of diagnoses across different diagnosis codes. The profile node can provide a histogram or other plot showing the distribution of diagnosis codes, which can help you identify any patterns or trends in the data.

In addition to monitoring the distribution of values, it can also be useful to monitor a list of values (LOV) deltas. LOV deltas refer to the difference between the list of values used in one dataset and the list of values used in another dataset. For example, if you are comparing a dataset of patient claims from one year to a dataset of patient claims from the previous year, you might be interested in the LOV deltas between the two datasets.

The DataOps Suite’s profile node can provide statistics on LOV deltas, which can help you identify any changes in the list of values used in your datasets. This can be useful for ensuring the quality and consistency of your data, and for identifying any potential issues or discrepancies.

As seen below 2 examples:

Example A deals with showcasing a change in the number of distinct values seen in a geography key of a patient claims dataset.

Example B showcases how the distribution of sales among different “Lines of Therapy” has been drastically changed indicating either an issue in the calculation of LOT, a change in behavior of the LOT in the drug in question, or worse a bug in the ETL.

Example B

Conclusion

In conclusion, data profiling is an important step in the data preparation process, and it is especially important in the pharmaceutical industry where data quality and integrity are critical. The DataOps Suite’s profile node is a powerful tool that can help you perform data profiling on your pharma datasets, and it can provide valuable insights and help you identify any potential issues or inconsistencies.

Some of the key features of the profile node include overview statistics, column statistics, and column distribution plots, which can all be useful in understanding the contents, structure, and quality of your data. In addition, the profile node can help you identify anomalies and outliers in your data, and it can provide statistics on LOV deltas, which can be useful for ensuring the consistency of your data.

Overall, the DataOps Suite’s profile node is a valuable tool that can help you ensure the quality and integrity of your pharma datasets and support more accurate and reliable analyses and modeling efforts.

Get a Free POC scheduled today!

Request Demo