Lead Pyspark Engineer

TCS • Owings Mills, MD

10d

About The Position

We are seeking a highly experienced Lead PySpark Engineer with over 10 years of experience in big data and distributed computing. The ideal candidate will have very strong hands-on experience with PySpark, Apache Spark, and Python. A strong command of SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.), proficiency in data modeling and ETL workflows, and familiarity with workflow schedulers like Airflow are essential. Experience with AWS cloud-based data platforms is required. This role involves leading the design, development, and deployment of PySpark-based big data solutions, architecting and optimizing ETL pipelines, and collaborating with various teams to deliver scalable solutions. The position also requires optimizing Spark performance, implementing data engineering best practices, and ensuring data security and compliance. Mentoring junior developers and code review are also key responsibilities.

Requirements

• 10+ years of experience in big data and distributed computing.
• Very Strong hands-on experience with PySpark, Apache Spark, and Python.
• Strong Hands on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).
• Proficiency in data modeling and ETL workflows.
• Proficiency with workflow schedulers like Airflow
• Hands on experience with AWS cloud-based data platforms.
• Strong problem-solving skills and ability to lead a team

Nice To Haves

• Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.

Responsibilities

• Lead the design, development, and deployment of PySpark-based big data solutions.
• Architect and optimize ETL pipelines for structured and unstructured data.
• Collaborate with Client, data engineers, data scientists, and business teams to understand requirements and provide scalable solutions.
• Optimize Spark performance through partitioning, caching, and tuning.
• Implement best practices in data engineering (CI/CD, version control, unit testing).
• Work with cloud platforms like AWS
• Ensure data security, governance, and compliance.
• Mentor junior developers and review code for best practices and efficiency

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed