Why ETL Validator when there is Python?

Why-ETL-Validator-when-there-is-Python_-17

ETL Testing Automation using python

I am not sure if you ever wrote Python but it is one of the coolest language out there. Easy to learn, easy to write and do yeah, easy to read! Writing code in Python is something I enjoy when I want to get a break from my routine job of putting together slides or preparing for upcoming demos.

Over the last few years, the usage of Python has gone up drastically and one such area is testing automation. With very few lines of code, you can achieve remarkable things. As an example, sometime back I had to compare the data in two CSV files (tens of thousands or rows)  and then spit out the differences. The code looked somewhat like this:

Automate ETL Testing using python script

#Python 2.7.5
orderFile = open(‘Source.csv’,’r’)
customOrderFile = open(‘Target.csv’,’r’)
orderLines = orderFile.readlines()
customOrderLines = customOrderFile.readlines()
orderFile.close();
customOrderFile.close();
orderSet = set(orderLines);
customOrderSet = set(customOrderLines);
print orderSet – customOrderSet;
#Response is below. Exactly, what I wanted to see.
set([‘492023-13942389n’, ‘492023-139584n’, ‘492023-139325n’, ‘492023-13939n’, ‘492023-13945n’, ‘492023-1394545n’])

Very simple, right? For a one-off, writing a script in Perl, Python or whatever your favorite language is fine. However, it is unfair to expect QA Analysts, Engineers to write scripts for their regular job. Few problems that often surface are:

  1. There is already enough work to do. Not many want to learn something new.
  2. There may be no common framework: The version of the scripting language may be different on each user’s machine or worse yet, each user may prefer to write scripts in languages of their choice.
  3. Let’s say we want to do regression testing on a bi-weekly basis (at the end of every sprint), how can we schedule them? Write another script? 🙂
  4. When we want to connect different databases (Netezza etc), we need specific drivers for each type. Painful!
  5. How do you get reporting on top of the results?
  6. How can you notify all the stakeholders?

Answers are not easy! So, to the extent possible, it is always better to have a consistent framework across all the business units within an organization to solve data testing challenges. ETL Validator provides you exactly what you need without writing code. Check it out! You will love it.

Datagaps-logo-1536x406-1

Established in the year 2010 with the mission of building trust in enterprise data & reports. Datagaps provides software for ETL Data Automation, Data Synchronization, Data Quality, Data Transformation, Test Data Generation, & BI Test Automation. An innovative company focused on providing the highest customer satisfaction. We are passionate about data-driven test automation. Our flagship solutions, ETL ValidatorDataFlow, and BI Validator are designed to help customers automate the testing of ETL, BI, Database, Data Lake, Flat File, & XML Data Sources. Our tools support Snowflake, Tableau, Amazon Redshift, Oracle Analytics, Salesforce, Microsoft Power BI, Azure Synapse, SAP BusinessObjects, IBM Cognos, etc., data warehousing projects, and BI platforms.  www.datagaps.com 

Related Posts:
Download Datasheet
Download Datasheet
Download Datasheet
Download Datasheet
Download Datasheet

Data Quality

Automate testing of Business Intelligence applications by making use of the metadata available from the BI tools such as Tableau, OBIEE, and Business Objects.

Synthetic Data

Automate testing of Business Intelligence applications by making use of the metadata available from the BI tools such as Tableau, OBIEE, and Business Objects.

ETL Testing

Automate testing of Business Intelligence applications by making use of the metadata available from the BI tools such as Tableau, OBIEE, and Business Objects.

BI Validation

Automate testing of Business Intelligence applications by making use of the metadata available from the BI tools such as Tableau, OBIEE, and Business Objects.
Products

product_menu_icon01

DataOps Suite

End-to-End Data Testing Automation

product_menu_icon02

ETL Validator

Automate your Data Reconciliation & ETL/ELT testing

product_menu_icon03

BI Validator

Automate functional regression & performance testing of BI reports

product_menu_icon04

DQ Monitor

Monitor quality of data being Ingested or at rest using DQ rules & AI

product_menu_icon05

Test Data Manager

Maintain data privacy by generating realistic synthetic data using AI

About
Free Trial