Lead Data Engineer - Python/Scala/Spark

We are seeking an experienced and dynamic Lead Data Engineer to join our growing data team. The ideal candidate will have 6-10 years of progressive experience in data engineering, with strong technical expertise in designing and maintaining complex data pipelines. You will lead a team of data engineers, collaborate across departments, and guide technical decision-making for scalable, reliable, and highperformance data solutions. Key Responsibilities : - Lead the design, development, and optimization of scalable data pipelines using technologies such as Python, SQL, Spark, Scala, and Airflow. - Oversee the management of cloud-based data infrastructure on AWS (S3, Redshift, Athena, Glue, EMR) to ensure high performance and availability. - Drive real-time data streaming initiatives using Kafka and Kinesis, ensuring reliability and low-latency performance. - Mentor and guide team members through code reviews, technical training, and best practices. - Collaborate with cross-functional teams to define requirements, set technical priorities, and ensure timely delivery of data solutions. - Provide on-call support, effectively triage incidents, and ensure rapid resolution. - Lead the adoption of industry best practices in data engineering, data governance, and system performance. - Ensure comprehensive documentation and clear communication across technical teams and stakeholders. - Drive continuous improvement in data quality, system performance, and operational processes. Required Skills & Experience : - 6-10 years of hands-on experience in data engineering with at least 2 years in a leadership capacity. - Proficiency in Scala, Python, Spark, and Airflow for building data pipelines. - Strong expertise in managing AWS data environments (Redshift, S3, Athena, Glue, EMR). - Experience with real-time streaming platforms such as Kafka and Kinesis. - Proven ability to manage on-call support operations and incident resolution. - Excellent communication and leadership skills with the ability to mentor and guide team members. - Strong problem-solving skills and ability to work in a fast-paced environment. Preferred Skills : - Advanced knowledge in Java, Vertx, and Spring Boot. - Experience with MySQL, Redshift, and other relational and NoSQL databases. - Exposure to cloud services and DevOps practices. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/12/2025