ETL Test Lead - PyTest

Responsibilities : - Develop comprehensive test plans, test cases, and test scripts based on business requirements and technical specifications. - Define test data requirements and create test data sets for various scenarios. - Establish testing methodologies and standards for ETL processes involving Ab Initio and PySpark. - Collaborate with developers, data engineers, and business analysts to understand requirements and ensure test coverage. - Execute test cases and scripts for Ab Initio graphs and PySpark jobs. - Perform data validation, data quality checks, and data integrity testing. - Identify, document, and track defects using bug tracking tools. - Analyze test results and provide detailed reports on findings. - Conduct performance testing and identify performance bottlenecks. - Perform regression testing, system integration testing, and user acceptance testing. - Validate Ab Initio graphs, components, and data flows. - Test Ab Initio parameters, variables, and metadata. - Verify data transformations and data mappings within Ab Initio. - Test Ab Initio performance and scalability. - Validate PySpark data transformations and data processing logic. - Test PySpark data frames, RDDs, and SQL queries. - Verify data consistency and accuracy in PySpark jobs. - Test PySpark performance and resource utilization. - Automation and Continuous Integration : - Develop and maintain automated test scripts for ETL processes. - Integrate testing into the CI/CD pipeline. - Utilize testing frameworks and tools to enhance test efficiency. - Automate data quality checks. - Work closely with development and data engineering teams to resolve defects. - Communicate test results and findings to stakeholders. - Participate in code reviews and provide feedback on testability. - Document test procedures and test results. Required Skills and Qualifications : - Bachelor's degree in Computer Science, Information Technology, or a related field. - Minimum 4 years of experience in ETL testing and data warehousing. - Strong proficiency in Ab Initio, including graph development and testing. - Hands-on experience with PySpark for data processing and testing. - Solid understanding of SQL and database concepts. - Experience with data quality tools and techniques. - Familiarity with testing frameworks and automation tools (e.g., JUnit, PyTest). - Experience with bug tracking tools (e.g., Jira, Azure DevOps). - Knowledge of data warehousing concepts and data modeling. - Strong analytical and problem-solving skills. - Excellent communication and collaboration skills. - - Experience with cloud platforms like AWS, Azure, or GCP is a plus. - - Experience with version control systems like Git. Preferred Qualifications : - Experience with other big data technologies (e.g., Hadoop, Hive, Kafka). - Knowledge of data governance and data security principles. - Certification in Ab Initio or PySpark. - Experience with data virtualization. (ref:hirist.tech)

Location: pune, IN

Posted Date: 5/6/2025