AWS Consultant - Python/PySpark

AWS CONSULTANT Mandatory Skills : Data pipeline, Aws Sagemaker, Aws Glue, aws, python, Aws Lambda, Amazon Ec2, deployment, Athena, devops, ansible, web application, Aws Cloud, Aws Cloudformation, Terraform, Boto3, Code Pipeline, Code Deploy Skill to Evaluate : AWS, Data PipeLine ,Cloud Infra,Glue,Python, Pyspark, Athena, ETL, DevOps, Code Pipeline Experience : 4 to 6 Years Location : Bengaluru Job Description : We are seeking an experienced Data Pipeline & Cloud Infrastructure Engineer to join our team. The ideal candidate will be responsible for building and maintaining robust data pipelines, managing cloud infrastructure (primarily EC2 and S3), supporting machine learning models, and ensuring smooth operations for web applications and analytics systems. You will work closely with data scientists and various teams to resolve issues, handle deployments, and maintain a secure environment AWS Consultant Roles & Responsibilities : Key Responsibilities : 1. Data Pipeline Development & Maintenance : - Build and maintain data pipelines for data enrichment and loading into base tables for ML model integration. - Leverage AWS services (Glue, Athena, S3) and Python (90%) PySpark (10%) to ensure efficient data flow. - Implement error handling and alerts with appropriate notifications. - Collaborate with SRI Data Scientists to troubleshoot and resolve data- related issues. 2. Workload Management : - Maintain EC2 workloads, ensuring security, setting up alerts, managing Linux environments, and handling deployments for machine learning models. 3. Web Application Hosting : - Host web applications on EC2 for analytics purposes using Python and the Streamlit framework. - Manage Linux-based EC2 instances, Elastic Load Balancers (ELB), and SSL/TLS certificates. 4. Reporting & Monitoring : - Generate and maintain various reports including billing, bill consolidation, data stats, and daily activity reports. 5. Adhoc Research Workloads : - Provision EC2 instances for short-term, ad-hoc research tasks by external users (e.g., interns). - Manage the lifecycle of such instances, including provisioning, access, and decommissioning. 6. Sage maker User Management : - Add users and manage domains within AWS SageMaker (permissions and infrastructure support). - No direct workload management but provide support related to infrastructure issues (e.g., access permissions). 7. Network & Access Management : - Manage Virtual Private Clouds (VPCs) and access controls. - Provide Sony VPN access on an ad-hoc basis and handle IP whitelisting requests. 8. CI/CD & Infrastructure Deployment : - Maintain CloudFormation (CFN) templates and deploy infrastructure for machine learning workflows. - Manage the version control of the ML codebase through Git. Qualifications : - Strong experience with AWS services (Glue, Athena, S3, EC2, VPCs). - Proficiency in Python (90%) and PySpark (10%) for data processing and automation. - Experience with web application hosting, particularly with Python/Streamlit framework. - Familiarity with CI/CD pipelines, CloudFormation, and version control (Git). - Experience with machine learning infrastructure and tools, particularly SageMaker, is a plus. - Strong troubleshooting skills and the ability to collaborate with cross-functional teams. Preferred Skills : - Linux system administration experience. - Experience with security best practices, monitoring, and alerting on cloud infrastructure. - Knowledge of networking concepts (VPCs, VPN, IP whitelisting). - Experience managing billing and report generation tasks. Additional Information : - Flexible working hours based on project deadlines. - Collaborative team environment with an opportunity to work with data scientists and engineers across various domains. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 6/5/2025