Success Story

Ntokoto Technology Group strives to provide high quality integrated and innovative solutions to their clients through mastering the latest technology, best practices, and recognized industry standards. As a QuerySurge partner, they are skilled in the implementation of QuerySurge and help clients fully realize the benefits of an automated data testing solution. When they were brought onto a project with a South African utility company that was struggling with data quality issues, they were able to assist them in a major ETL process overhaul while validating all data at every integration point. 

Background

A major utility company suspected there were quality issues within their data. They worked with QuerySurge partner Ntokoto Technology Group (NTG) to design and execute a full data reconciliation test. QuerySurge identified several data issues, which prompted stakeholders to reconsider their ETL process. A decision to implement daily ETL load checks was made and NTG, with the help of QuerySurge, was able to design and implement tests which validated 100% of the company’s data. 

The ETL process incorporated a variety of technologies and data sources such as SAP, Cloudera HDFS, Teradata, Excel, and flat files. The time and cost associated with their existing procedures led them to pursue a direct ETL load from SAP to Teradata., bypassing data loads into Cloudera. This was accompanied by a restructuring of their data pipelines to migrate off Informatica and onto a SSIS-based solution.

Challenges

Prior to engaging Ntokoto Technology Group, data validations were limited to Excel macros and data sampling tests. The limitation in coverage extended test cycle times, which led to missed deadlines and incidents of bad data in production. This type of testing methodology was prohibitive to a continuous testing model and could not be easily integrated into their CI/CD process.

Additionally, the ETL process was loading full data sets with each iteration. It often took many hours for the entire procedure to complete, which delayed validation efforts. This made it especially difficult to complete testing in a reasonable timeframe, especially when code was being promoted thorough the environments. Data validations in DEV, QA, UAT and Production environments were lacking in coverage and often delayed.

Solution

The team from NTG was able to work with the utility company to develop a new load process that helped improve the overall testing effort. By eliminating Cloudera from the architecture and transitioning over to an SSIS solution, data was able to move directly from SAP into Teradata. The full data loads were then pared down into incremental loads, which greatly reduced the ETL run time and allowed for targeted testing of new data loads. 

NTG developed a full suite of automated data tests using QuerySurge to validate the daily loads. To improve the extraction time from SAP, they implemented a staging area in Microsoft’s Azure SQL. This dramatically improved load time and further enhanced the speed of testing.

Utilizing QuerySurge’s Integration with Azure DevOps, the team was able to easily create Azure DevOps pipelines. This streamlined the deployment and testing processes across all environments and provided reusable tests for DEV and QA in UAT and Production. 

Benefits

NTG improved the utility company’s data testing coverage from only small sample sets to 100% data coverage. With an average table size of 1 million rows, QuerySurge was an integral part of this effort, providing an automated solution that validated their data at speed. Over 5,000 QueryPairs were developed to achieve this coverage and that number grows with each release as additional tests are added into the regression suite. The seamless integration into their CI/CD pipelines allows for reusability across environments and has streamlined testing for each release.

Additionally, NTG was able to set up QuerySurge email notifications to inform business users when each test was completed. Business users were particularly pleased as they were immediately made aware of any data objects that may have an issue. This was especially important as the Teradata source populated critical business intelligence reports that were being used in key decision-making efforts.

The initial project was focused on critical data for the Finance and Supply Chain departments within the company. After the successful implementation of QuerySurge testing led by the NTG team, the utility company is now looking to expand the coverage of this testing implementation enterprise wide

About Ntokoto Technology Group:

Ntokoto Technology Group (PTY) Ltd is an ICT company founded with the sole purpose of delivering tailored and innovative IT and telecommunications services that help organizations leverage technology, communications and people to: