Python is easy to write and fine for one-off ETL comparisons — diffing two CSVs takes a few lines of code. But this post argues it breaks down as a team-wide framework: inconsistent script versions, no shared language across testers, no built-in scheduling for recurring regression runs, and separate drivers needed per database connection. ETL Validator solves all of this without requiring anyone to write code.
Key Takeaways
- Python is great for one-off comparisons, not team-scale testing.
- No common framework — script versions and language preferences vary tester to tester.
- No built-in scheduling/reporting — recurring regression runs need extra custom code.
- Separate drivers per database add overhead that a unified tool removes.
ETL Testing Automation using python
I am not sure if you ever wrote Python but it is one of the coolest language out there. Easy to learn, easy to write and do yeah, easy to read! Writing code in Python is something I enjoy when I want to get a break from my routine job of putting together slides or preparing for upcoming demos.
Over the last few years, the usage of Python has gone up drastically and one such area is testing automation. With very few lines of code, you can achieve remarkable things. As an example, sometime back I had to compare the data in two CSV files (tens of thousands or rows) and then spit out the differences. The code looked somewhat like this:
Automate ETL Testing using python script
#Python 2.7.5
orderFile = open(‘Source.csv’,’r’)
customOrderFile = open(‘Target.csv’,’r’)
orderLines = orderFile.readlines()
customOrderLines = customOrderFile.readlines()
orderFile.close();
customOrderFile.close();
orderSet = set(orderLines);
customOrderSet = set(customOrderLines);
print orderSet – customOrderSet;
#Response is below. Exactly, what I wanted to see.
set([‘492023-13942389n’, ‘492023-139584n’, ‘492023-139325n’, ‘492023-13939n’, ‘492023-13945n’, ‘492023-1394545n’])
Very simple, right? For a one-off, writing a script in Perl, Python or whatever your favorite language is fine. However, it is unfair to expect QA Analysts, Engineers to write scripts for their regular job. Few problems that often surface are:
- There is already enough work to do. Not many want to learn something new.
- There may be no common framework: The version of the scripting language may be different on each user’s machine or worse yet, each user may prefer to write scripts in languages of their choice.
- Let’s say we want to do regression testing on a bi-weekly basis (at the end of every sprint), how can we schedule them? Write another script?
- When we want to connect different databases (Netezza etc), we need specific drivers for each type. Painful!
- How do you get reporting on top of the results?
- How can you notify all the stakeholders?
Answers are not easy! So, to the extent possible, it is always better to have a consistent framework across all the business units within an organization to solve data testing challenges. ETL Validator provides you exactly what you need without writing code. Check it out! You will love it.
Frequently Asked Questions: Python Scripts vs. ETL Validator for ETL Testing
1) Is Python good for ETL testing?
Great for quick one-off comparisons, harder to sustain at team scale or on a schedule.
2) What breaks when a QA team relies on custom scripts?
Version inconsistency, no shared language, no scheduling, separate drivers per DB.
3) How does ETL Validator differ from custom scripts?
No-code, consistent framework covering scheduling, connectivity, reporting, and notifications.
4) Why does a consistent framework matter org-wide?
It avoids fragmentation from individually-written scripts and keeps regression testing uniform across teams.
Rajesh Kumar A
Digital Marketing Manager, Datagaps
Digital Marketing Manager at Datagaps. Drives data-driven growth through content, performance campaigns, and marketing technology.
Subrahmanya Narayana Chirravuri
Senior Director, Technology, Datagaps
Senior Director of Technology at Datagaps. Leads engineering for the ETL, BI, and data-quality validation platforms.
