Hadoop Developer - Big Data Tools

Job Description: We are looking for an experienced Hadoop Developer to join our data engineering team. The ideal candidate should have strong expertise in Big Data technologies, including Hadoop, Spark, and related ecosystems, and should be capable of designing, developing, and optimizing large-scale data processing solutions. Roles and Responsibilities : - Develop and maintain Hadoop-based data pipelines to process large datasets efficiently. - Implement ETL processes using tools like Apache Hive, Apache Spark, Sqoop, and Pig. - Work with HDFS, HBase, and other Hadoop ecosystem components to store and manage structured and unstructured data. - Optimize Hadoop jobs for high performance and scalability. - Collaborate with data analysts, business teams, and other stakeholders to understand data requirements and translate them into efficient solutions. - Ensure data security and governance while handling sensitive information. - Monitor cluster performance and troubleshoot issues related to Hadoop infrastructure. - Develop real-time and batch data processing applications using Spark and Kafka. - Work with Cloud platforms (AWS, Azure, GCP) for Big Data processing and deployment. - Document technical specifications, best practices, and design guidelines. Required Skills : - Strong experience in Hadoop ecosystem (HDFS, Hive, HBase, Pig, Sqoop, Oozie). - Expertise in Apache Spark (batch and streaming) using Scala or Python. - Experience with SQL and NoSQL databases (MySQL, PostgreSQL, Cassandra, MongoDB). - Proficiency in Java, Scala, or Python for data processing. - Hands-on experience with Kafka, Flume, or other streaming technologies. - Good understanding of Data Warehousing concepts. - Experience working with cloud-based Big Data solutions (AWS EMR, Azure HDInsight, GCP Dataproc). - Familiarity with CI/CD pipelines and version control tools like Git. - Strong problem-solving and debugging skills. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/4/2025