Sr. Software Engineer (Python + PySpark)

Capgemini • Vancouver, BC

1d • Remote

About The Position

As a Senior Software Engineer, you will work on one of the world’s largest social media platforms, handling multiple petabytes of data processed daily. You will be part of a self-organized R&D team working in a challenging and innovative environment for our client. You will investigate, design, and implement solutions for complex technical challenges using cutting-edge technologies, including building and enhancing data processing platforms that power software used by hundreds of millions of users.

Requirements

• 5+ years of experience as a Software Engineer with strong Python development skills, including hands-on PySpark and Apache Spark implementation.
• Mid-level Java proficiency (minimum 2.5/5), with practical experience writing, debugging, and maintaining Java code.
• 3+ years of experience building, operating, and supporting large-scale, scalable Big Data platforms or infrastructure.
• Strong knowledge of Big Data technologies, including: Apache Spark, Hadoop ecosystem, Parquet / ORC file formats, Apache Flink (or similar stream-processing frameworks).
• Proven troubleshooting and production support skills, including the ability to handle SEV-1 / critical production incidents under pressure.
• Strong ability to organize, document, and communicate solutions, including technical design, root cause analysis (RCA), and operational runbooks.
• A passion for streamlining systems and processes to make the difficult trivial
• Strong OOP skills
• Effective communication (oral & written), collaboration, and interpersonal skills
• Result oriented approach

Nice To Haves

• Experience with Node.js.
• Experience with AWS, Kubernetes (K8s), and Docker.

Responsibilities

• Receive tasks from the Project Lead or Team Lead (TL), prepare functional and design specifications, and align them with all stakeholders.
• Ensure assigned areas are delivered on time and meet quality expectations.
• Provide effort estimations, agree on timelines with the manager, and contribute to project planning.
• Evaluate existing data systems and identify improvement opportunities.
• Develop best practices for data coding to ensure system-wide consistency.
• Update and optimize local and metadata models.
• Assess implemented data systems for discrepancies, variances, and performance efficiency.
• Troubleshoot and optimize data infrastructure and pipelines.
• Understand business drivers and analytical use cases, translating them into reliable data products.
• Design, implement, and maintain cloud-based pipelines that deliver business-critical data efficiently.
• Identify area-level risks and implement mitigation plans.
• Report readiness/quality status and escalate critical issues appropriately.
• Take responsibility for resolving crisis situations within the assigned area of responsibility (AOR).
• Initiate and conduct code reviews, and help establish coding standards, conventions, and guidelines.
• Propose technical and functional improvements to enhance the product.
• Continuously upgrade professional knowledge and skills.
• Collaborate closely with cross-functional teams.

Benefits

• Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed