Data Engineer/Scientist (Advanced) 1210

Essential skills requirements: Proficiency in Python 3.x for data processing and automation. Experience with AWS Glue for ETL processes. Strong knowledge of AWS Athena for querying large datasets. Hands-on experience with AWS Lambda for serverless computing. Familiarity with AWS EC2 for scalable computing resources. Expertise in AWS CloudWatch for monitoring and logging. Proficiency in working with PostgreSQL RDS for database management. Experience with AWS QuickSight for data visualization and reporting. Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs). Proficiency in Git and GitHub for version control and collaboration, including experience with GitHub CI/CD pipelines. Any additional responsibilities assigned in the Agile Working Model (AWM) Charter Advantageous skills requirements: Extensive experience with Terraform for infrastructure as code, including creating, managing, and optimizing AWS resources. Experience with other AWS services such as S3, Kinesis, and DynamoDB. Knowledge of data formats such as Parquet, AVRO, JSON, and CSV. Experience with Docker for containerization. Understanding of Big Data technologies and frameworks. Familiarity with Agile working models and tools like JIRA and Confluence. Experience with data quality tools such as Great Expectations. Knowledge of REST API development and integration. Strong analytical skills for troubleshooting and optimizing data pipelines. Experience in developing technical documentation and artefacts. What will be your role and responsibilities? Design, develop, and maintain data ingestion pipelines for BMWs connected fleet, focusing on Call Detail Records (CDRs). Utilize AWS Glue for ETL processes to transform and load data into the analytics platform. Implement efficient querying solutions using AWS Athena. Develop serverless applications and workflows using AWS Lambda. Monitor and optimize data pipelines using AWS CloudWatch. Manage and maintain PostgreSQL RDS databases for data storage and retrieval. Create interactive dashboards and reports using AWS QuickSight. Leverage Terraform extensively to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability. Use Git and GitHub for version control, collaboration, and maintaining code repositories. Implement and manage GitHub CI/CD pipelines to automate testing, deployment, and delivery processes. Ensure data security and compliance with BMWs information classification requirements. Stay updated with the latest data engineering tools, technologies, and industry trends. Identify opportunities for process improvements and automation to enhance pipeline efficiency. Collaborate with cross-functional teams to understand data requirements and deliver solutions. Develop and maintain technical documentation for data engineering processes and solutions.

Location: Menlyn, ZA

Posted Date: 6/14/2025