This step-by-step tutorial shows how to validate REST API data in DataOps Suite using DataFlow. It covers using the Code Component (Scala, Python, or SparkR) to fetch and parse API responses into a dataset, applying the Attribute Component to correct data types and column names, and using the Data Compare Component to validate the API dataset against a file-based dataset. The guide walks through mapping, key selection, running comparisons, and reviewing differences via DataFlowโs visual execution tracker.
Key Takeaways
- Code Component enables REST API ingestion โ supports Scala, Python, and SparkR, letting users write custom code to pull API responses and convert them into a queryable dataset within DataFlow.
- Attribute Component fixes data types post-ingestion โ since API data initially loads as string type, this component lets users rename columns and correctly assign data types before comparison.
- Data Compare Component validates API data against other sources โ supports flexible column mapping (by order or by name) and key-based comparison, surfacing duplicates, mismatches, and differences.
- DataFlow provides visual, reusable pipelines โ once built, the flow can be re-run anytime via โRun DataFlow,โ with color-coded status tracking (green/blue/yellow/red) for each componentโs execution state.
| DataFlow Component | Role in This Walkthrough |
|---|---|
| Code Component | Reads the REST API response (via Scala/Python/Spark SQL) and converts it into a dataset |
| File Component | Reads the comparison dataset from a file |
| Attribute Component | Converts data types and renames columns on the API-derived dataset before comparison |
| Data Compare Component | Compares the two datasets and reports duplicates, differences, and records unique to each side |
- On the left menu, select โDataFlowsโ.
- Click on โNew DataFlowโ button on the top right corner.
- In the โNew DataFlowโ dialog, fill the details and Save.
- Name: Any random name to identify DataFlow.
- Livy server: By default, the application comes with a default livy server. Select any configured livy server.
- During the first run, all the list of components are displayed (as shown in the below image). Select Code component in the processor bucket.
- A new Code component will be opened with the โPropertiesโ tab selected. Fill the details and then go to the next step โ
- Name: Any random name.
- Dependency: Not required as it is the first component.
- Description: Optional. You can give any useful info about the code component.
- Dataset name: Name of the dataset which you want to create using the code. You can give multiple names, separated by commas.
- Select the kind. Scala is selected by default. The Code component supports Scala, Python and SparkR languages. By clicking on the โSample API codeโ button on the top right, a sample code will be populated to read data from Rest API. Here a sample code is provided.
Code:
import spark.implicits._;
var jsonStr =scala.io.Source.fromURL("http://192.168.6.42:9080/DataPrepRest/api/v1.0/templates/table?containerId=81&userName=sh&password=******&url=jdbc:oracle:thin:@192.168.6.76:1521:orcl&schema=sh&table=customers").mkString;
var df = spark.read.json(Seq(jsonStr).toDS());
df.createOrReplaceTempView("code_ds");
df.cache();
After execution of the above code a dataset with the name code_ds will be created. You can write multiple sets of such codes and can create multiple datasets. These datasets should be listed in the Properties tab as mentioned in the 5th step.
- Create a new File component. Fill the details and move to the next step โ
- Name: Any random name.
- Data Source: File data source list can be seen here. You need to select a data source.
- Dependency: In the present case, thereโs no need to give any dependency.
- Description: Optional. Write some basic info about the component.
- Dataset name: For File components, only one dataset will be created. Default name will be populated based on the component name. You can enter your desired Dataset name.
- In the File step, fill in the details as shown below โ
- File Name: Give the filename you want to read. Enter the filename manually or select a filename in the Files panel on the right.
- Encode: Optional (File encoding type).
- Options: Spark file read options. Some important options will be popped up with the default values. Please go through the following link for further info โ
https://docs.databricks.com/data/data-sources/read-csv.html.
- code_ds is the dataset created by reading Rest API data in the Code component. By default, all the data types will be considered as string. If you want to change these data types or column names, you can use the Attribute component. For any dataset, you can change the data types by using the Attribute component. Create a new Attribute component by using Add component.
Fill the details โ
- Name: Any random name
- Source Dataset: For which dataset user wants to change data types and column names. In the current example, we are selecting code_ds which is the output of the Code component.
- Dependency: As this component can be run only after creation of code_ds dataset from the Code component, you must add the code component in the dependency list.
- Description: Optional (description of the component).
- Dataset Name: Output dataset name. After converting data types and column names, a new dataset will be created as the output. Default is the component name.
- In the rename step, enter the desired column names and data types.
- Save and Run the component.
- Now click on Add component and select โData Compareโ โ a data reconciliation capability โ from the โData Quality bucketโ as shown below โ
- In the โData Compareโ component, you need to give two datasets as input. The comparison would be between these two datasets. Fill the details โ
- Name: Any Random name.
- Dataset A: In this example we are selecting the output of Attribute component.
- Dataset B: In this example we are selecting output of File component.
- Dependency: In this example we are consuming datasets from file component and attribute component. So give both of them as dependencies.
- Compare type: Different comparison types you want to run.
- Description: Description of the component.
- Dataset Name: Default name will be populated.
- In the mapping step both dataset A and dataset B will be mapped by order of columns by default. If required, you can click on the โRemap by Nameโ button to reorder the mapping. You can select unique keys, then comparison would be based on keys. It allows multiple key columns. Move to next step after changes done.
- Run the component. Each component executes in a set of statements. Data Comparison component contains more number of statements and the progress of execution can be seen at the bottom in the โRunโ tab. After the Run is completed, you can see the component results as shown in the following images. At the bottom, you can see failed and passed statements. These statements contain Duplicate calculation, Only in Dataset A, Only in Dataset B, Differences etc. For each calculation a statement will be there. By clicking on the link, you can see the details of each statement.
- By clicking on the difference count statement, a window pops up. Please check the following image.
- Now the design of DataFlow has been completed. Itโs a one time step. You can run the DataFlow whenever you want just by clicking the โRun DataFlowโ button at the top of the DataFlow window. Then the following window opens โ
This image is created based on the dependencies given. And it is the execution order of components. Here each color indicates the progress of different components โ
- Green: Successfully completed and the status is passed.
- Blue: In Queue.
- Yellow: Running
- Red: Completed. But the status is failure.
Once this DataFlow is built, validating a REST API response against a file (or any other dataset) becomes a one-click, repeatable check โ you donโt need to rebuild the Code, File, Attribute, and Data Compare components each time, just re-run the DataFlow and watch the color-coded status of each step.
Conclusion
As the MuleSoft data shows, most organizations still struggle to confirm that their โconnectedโ systems actually agree with each other โ and REST APIs, often the connective tissue between modern applications, are no exception. This walkthrough demonstrates that validating API responses doesnโt have to mean writing one-off scripts every time: by combining the Code Component to ingest the API response, the Attribute Component to correctly type and rename fields, and the Data Compare Component to reconcile it against a trusted dataset, DataOps Suite turns API validation into a reusable, visual DataFlow rather than a manual, repeat-every-time task. Once built, the same pipeline can be re-run on demand, giving teams a fast, repeatable way to confirm their API data is accurate โ not just that the endpoint responded.
FAQs: REST API Data Validation in DataOps Suite
1) How do you pull REST API data into DataOps Suite?
Using the Code Component, which supports Scala, Python, and SparkR, you can write custom code to call a REST API, parse the response, and convert it into a dataset that can be used within a DataFlow.
2) Why is the Attribute Component needed after fetching API data?
REST API data is initially loaded with all columns as string data types. The Attribute Component allows users to rename columns and assign the correct data typesโsuch as integer, date, decimal, or booleanโbefore performing comparisons or downstream processing.
3) How do you compare REST API data against another data source?
The Data Compare Component validates the API dataset against a second dataset, such as a file or database table. Users can map columns by name or position and define key columns to accurately identify matching records and detect differences.
4) How can you track whether a DataFlow ran successfully?
DataFlow provides color-coded execution status indicators for every component: green for successful execution, blue for in progress, yellow for warnings, and red for errors, making it easy to monitor workflow execution.
Rajesh Kumar A
Digital Marketing Manager, Datagaps
Digital Marketing Manager at Datagaps. Drives data-driven growth through content, performance campaigns, and marketing technology.
Subrahmanya Narayana Chirravuri
Senior Director, Technology, Datagaps
Senior Director of Technology at Datagaps. Leads engineering for the ETL, BI, and data-quality validation platforms.
