Big Data Engineer

HEXAWARE • United States,

15h

About The Position

We are seeking a Senior Level Python developer with 8 years of software development experience to join our team as a Big Data Engineer. In this role, you will be responsible for building and maintaining ETL pipelines, orchestrating workflows, implementing messaging and event-driven integrations, and designing/optimizing data storage and querying solutions. You will also write efficient SQL, integrate data from APIs, process JSON data, implement data quality checks, and participate in CI/CD practices. Collaboration with cross-functional teams to translate business requirements into technical solutions is a key aspect of this position.

Requirements

• 4-8 years of software development experience across the appropriate platform.
• Strong hands-on experience with Python, PySpark, API’s and SQL.
• Experience with ETL/data pipeline development and Orchestration using Step functions / AirFlow.
• Working knowledge of AWS services including Glue, Lambda, Step Functions, Redshift, S3, SNS, and SQS.
• Experience with Athena, EMR, Kinesis, DynamoDB, or RDS.
• Good Knowledge on CloudWatch, logging, and production support.
• Understanding of data warehousing, data lakes, Lake House and query optimization.
• Experience with GitLab/Terraform or similar and CI/CD workflows.
• Good understanding of using AI tools like Github Copilot or similar for code productivity
• Exposure to enterprise data lake or cloud migration initiatives.
• Have an eye to solving complex problems, great communication with stakeholders
• Have a good understanding of performance engineering of code pipelines and near real time systems
• Good understanding on Agents and MCP

Responsibilities

• Build and maintain ETL pipelines using Python and PySpark on AWS Glue and related platforms.
• Orchestrate workflows using AWS Step Functions and Lambda.
• Implement messaging and event-driven integrations using SNS and SQS.
• Design and optimize storage and querying solutions in Amazon Redshift, RDS, Oracle and S3-based architectures.
• Write efficient SQL for transformations, validation, and reporting.
• Integrate data from APIs and process structured and semi-structured JSON data.
• Implement data quality checks, monitoring, and operational support processes.
• Participate in CI/CD and version control practices for deployment and release management.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed