Lead Data Lakehouse Architect - Apache Spark

Job Description : Provide technical leadership and architectural guidance : - Provide technical leadership and architectural guidance for the design and implementation of our open Data Lakehouse strategy, with Apache Iceberg as the core. Define and champion best practices : - Define and champion best practices for Data Lakehouse architecture, leveraging Apache Iceberg and Trino for optimal data querying, analysis, and governance at scale. Lead the development and management of the Lakehouse : - Lead the development and management of the Lakehouse for massive data platforms, encompassing all critical aspects: storage strategies, efficient computing frameworks, robust data ingestion pipelines, comprehensive data governance models, effective data management practices, and stringent performance optimization techniques. Leverage your deep expertise : - Leverage your deep expertise in distributed computing technologies, particularly Apache Spark, to architect and optimize data processing and transformation workflows within the Lakehouse. Apply your extensive knowledge : - Apply your extensive knowledge of diverse data platforms, scalable data lakes, and high-throughput data ingestion systems to build a unified and efficient data ecosystem. Take ownership : - Take ownership of identifying, diagnosing, and resolving intricate technical challenges within our large-scale distributed data infrastructure. Collaborate closely : - Collaborate closely with cross-functional teams, including data engineers, data scientists, and business stakeholders, to understand their evolving data needs and translate them into strategic architectural blueprints. Drive the adoption : - Drive the adoption of data governance frameworks and ensure data quality, security, and compliance within the Lakehouse environment. Continuously evaluate : - Continuously evaluate emerging technologies and industry trends to identify opportunities for innovation and improvement within our data platform. Mentor and guide : - Mentor and guide other data architects and engineers, fostering a culture of technical excellence and knowledge Skills : Key Qualifications : - Extensive and deep experience with open Data Lakehouse concepts, architecture, and implementation, with a strong emphasis on Apache Iceberg. - Proven expertise in designing and optimizing Data Lakehouse architectures utilizing Apache Iceberg and Trino for high-performance data querying and analysis. - Significant and demonstrable experience in leading the building and management of Lakehouse solutions for very large-scale data platforms, covering storage, computing, data ingestion, data governance, data management, and performance optimization. - Expert-level experience with distributed computing technology(s), particularly Apache Spark, including performance tuning and optimization. - Comprehensive experience with a variety of data platforms, scalable data lakes, and high-volume data ingestion systems that operate reliably at scale. - Exceptional skills in troubleshooting and resolving complex, high-impact issues in large-scale distributed data systems. Good-to-Have Skills : - Experience with other data lake formats (e.g., Delta Lake, Hudi) and their integration with the Lakehouse. - Deep understanding of cloud-based data warehousing and data lake services (e.g., AWS S3, Azure Data Lake Storage, Google Cloud Storage, AWS Glue, Azure Data Factory, Google Cloud Dataflow). - Experience implementing and managing data governance tools and frameworks within a Lakehouse environment. - Strong knowledge of data security best practices and their application in a Lakehouse architecture. - Experience with performance monitoring and tuning tools specifically for distributed data systems. - Familiarity with data cataloging and metadata management solutions (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/9/2025