Big Data Lead

HEXAWARE • United States,

5d

About The Position

This role focuses on the development and maintenance of data pipelines, ensuring data quality and integrity, and collaborating with stakeholders to deliver technical solutions. The position requires strong coding proficiency in Python and Pyspark, experience with big data frameworks and cloud platforms, and a solid understanding of database design and data warehousing.

Requirements

• Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred)
• Experience with big data frameworks (Hadoop, Spark).
• Experience with cloud platforms ( AWS, Azure or GCP)
• Experience with Code versioning tools ( Bitbucket, Github )
• Experience with CI/CD and setting up pipelines.
• Solid understanding of database design principles, data modelling, schemas and data warehousing solutions.
• Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.

Responsibilities

• Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
• Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement.
• Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle.
• Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions.
• Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability.
• Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed