Case Study
Expleo executes petabyte-sized complex
data migration using QuerySurge
Expleo executes petabyte-sized complex
data migration using QuerySurge
Summary
Expleo provided data validation services for Union Bank of the Philippines (UBP) during a complex migration project encompassing over 15 applications and 1.7 Petabytes of data. As a Platinum Alliance partner of QuerySurge, Expleo was well positioned to utilize QuerySurge’s comprehensive feature set to help streamline the validation efforts. With Expleo’s expertise, UBP successfully navigated the complexities of data validation in this large-scale migration project. The robust features and capabilities of QuerySurge not only optimized validation processes but also resulted in substantial cost savings and improved data quality, ultimately ensuring the project’s overall success.
Background
Following a bank acquisition, Union Bank of the Philippines (UBP) sought to integrate all acquired data into its systems. The data migration project scope was extensive, encompassing over 15 business-critical applications utilizing various technologies. The datasets, totaling more than 1.7 Petabytes, comprised structured and unstructured data sourced from leading technologies such as the AWS technology stack, Oracle, Postgres, MySQL, and Microsoft’s SQL Server.
The primary objective of the migration project was to seamlessly incorporate these heterogeneous data sources into UBP’s systems. Given the sensitive nature of the data, meticulous validation processes were imperative for the Expleo team. Achieving 100% validation coverage for all migrated data was essential for project success.
Challenges
QuerySurge and Expleo’s expertise helped the bank over complex challenges. Initial validation attempts using methods like minus queries or Excel macros required extensive manual effort, leading to cumbersome and error-prone processes that hindered project efficiency. The sheer size of the data sets being compared presented another challenge. Reviewing the results sets to identify failures consumed considerable time for team members.
Additionally, the diversity of databases further complicated testing efforts, requiring the creation of connections to different environments. This complexity added to the challenges of the validation process. After some initial testing cycles, it became evident that meeting the project’s timeline would be unattainable using these manual methods.
Solution
Expleo leveraged QuerySurge’s platform to streamline validation by facilitating seamless data comparison between source and target databases. QuerySurge’s ability to connect to all data sources in scope and consolidate them into a single solution significantly improved efficiency.
The advanced scheduling features of QuerySurge allowed team members to schedule runs instantly, enabling continuous testing around the clock. The feature ensured that time limitations did not constrain validation efforts.
QuerySurge’s reporting and failure analysis functionalities enhanced overall efficiency and effectiveness in validation efforts. These features enabled team members to identify complex fields and quickly alert the appropriate resources to resolve any identified issues. By leveraging QuerySurge’s robust capabilities, Expleo efficiently handled UBP’s large data volumes, ensuring thorough validation of all data sets.
Average table size validation: 2 GB
Largest table size validated: 450 Millon Records
Total tests created: 25,000 QueryPairs (Source to Target Verifications)
QuerySurge server specifications:
- Processor Cores: 8
- RAM: 32 GB
- Agents: 10
QuerySurge’s insightful reporting enabled us to thoroughly analyze the entirety of the data.”
Benefits
The adoption of QuerySurge yielded numerous benefits for UBP’s data migration. The Expleo team improved efficiency and accuracy in data validation, enabling better decision-making throughout the migration project lifecycle. The team successfully completed the migration with 100% data verification for all 1.7 Petabytes of data. Cost savings were achieved through optimized resource allocation and reduced manual validation efforts, resulting in enhanced operational efficiency.
Breaking down the comparison of manual effort versus the effort with QuerySurge, the results are as follows:
| Manual Testing Execution Estimate | Automated Testing with QuerySurge | |
|---|---|---|
| Total Time | 25,000 hours = 3,125 days | 9,250 hours = 1,156 days |
| Number of Resources | 20 | 10 |
| Total Time/Resources | 3125/20 = 156 days (approx.) | 1156/10 = 116 days (approx.) |
| Total cost | $979,500 | $429,251 |
By automating the testing process with QuerySurge, Expleo helped the bank achieve significant savings and a Return on Investment (ROI). Reduced manual effort and human error led to faster and more reliable results. The total estimated savings amounted to $550,249, representing an impressive ROI of 228%.
Using this solution helped us pinpoint any disparities between the source and the target.”
About Expleo:
Expleo is a global engineering, technology and consulting service provider that partners with leading organizations to guide them through their business transformation, helping them achieve operational excellence and future-proof their businesses. Expleo benefits from more than 50 years of experience developing complex products, optimizing manufacturing processes, and ensuring the quality of information systems. Leveraging its deep sector knowledge and wide-ranging expertise in fields including AI engineering, digitalization, hyperautomation, cybersecurity and data science, the group’s mission is to fast-track innovation through each step of the value chain. As a responsible and diverse organization, Expleo is committed to doing business with integrity and working towards a more sustainable and secure society. Expleo boasts an extensive global footprint, powered by 19,000 highly skilled experts delivering value in 30 countries.
About RTTS & QuerySurge:
RTTS, the developer of QuerySurge, is the premier pure-play QA & Testing organization that specializes in test automation. Headquartered in New York, RTTS has had 1,000+ successful engagements at over 700 corporations since 1996.
QuerySurge is RTTS’ smart data testing solution that automates the data validation and ETL testing of Big Data, Data Warehouses, Business Intelligence Reports and Enterprise Applications with full DevOps/DataOps functionality for continuous testing.
