Lead Data Engineer - ETL/Big Data

Role Summary : We are seeking an experienced Senior Data Engineer to join our dynamic team. You will be responsible for designing, implementing, and maintaining scalable data pipelines and infrastructure to support our data processing needs. This role requires a deep understanding of data architecture, ETL processes, and big data technologies. The ideal candidate will have strong problem-solving skills and a passion for building robust and ecient data systems. Key Responsibilities : - Design, develop, and maintain robust data pipelines to ingest, process and transform large volumes of structured and unstructured data from diverse sources. - Design and implement data models that meet the business requirements for analytics, and reporting applications. Optimize data models for performance and scalability. - Develop : Extract, Transform, Load (ETL) processes to cleanse, enrich, and aggregate raw data into usable formats for analysis and reporting. - Integrate data from various internal and external sources, ensuring data quality, consistency, and integrity throughout the process. - Identify and address performance bottlenecks in data pipelines and queries. Optimize data processing workflows for speed, e ciency, and reliability. - Design and maintain data warehouse solutions for storing and querying structured data. Implement best practices for data partitioning, indexing, and compression. - Monitor data pipelines and systems for errors, anomalies, and performance issues. Develop and implement proactive measures to ensure data quality and system reliability. - Document data pipelines, data models, and system architectures. Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver solutions. Requirements : - 2-3 years of experience as a Data Engineer, with a focus on building and maintaining large-scale data infrastructure. - Proficiency in programming languages such as Python and Java. - Strong SQL skills and experience with relational databases (e.g., PostgreSQL, MySQL). - Experience with NoSQL databases such as MongoDB and DynamoDB. - Familiarity with cloud platforms such as AWS. - Solid understanding of data warehousing concepts and architectures. - Experience with data modelling, ETL processes, and data integration techniques. - Excellent problem-solving skills and attention to detail. - Strong communication and collaboration skills. - Bachelor's or master's degree in computer science, Engineering, or a related field. Mandatory Skills : - Experience with Tool/Technologies Apache Airflow, AWS Athena, Kafka, Python, Bash, AWS S3, Metabase or any data visualisation tool. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/4/2025