Spark Engineer
About The Position
Our financial services client, within a large US bank with international presence, oversees Data Pipelines and Automation Services within a Chief Data Office. The team is responsible for enterprise data platform modernization initiatives spanning data movement, automation, platform engineering, and operational transformation. The client is responsible for enterprise ETL platforms, data pipeline frameworks, and automation services that enable secure, governed, and scalable data delivery across hybrid cloud environments. The team is executing strategic modernization programs including legacy platform retirement, cloud-native platform adoption, containerization, CI/CD implementation, and operational process automation. The team for this project is leading the development of a greenfield, best-of-breed, next-generation enterprise data pipeline platform that unifies data engineering standards, governance, lineage, and deployment across multiple data processing technologies and environments. The project team is building a platform to unify data pipeline execution across the enterprise and leading a greenfield platform that requires, for engineer roles on the team, a strong understanding of APIs, Python, Sparkflow, Docker, K8s, Automated CI/CD with Github Actions or other tools, composable services, data governance, and distributed systems. The goal is to turn complex systems into clean, reusable services via Python.
Requirements
- Experience with Apache Spark, Spark Engine, Spark Flow
- Familiarity with big data ecosystems: Hadoop, Hive, SQL, Kafka
- Exposure to cloud platforms: AWS, Azure, or GCP
- Hands-on experience building and integrating APIs and microservices
- Strong problem-solving and debugging skills
Responsibilities
- Design and develop scalable integrations with Spark Engine and Spark Flow for the Unity platform
- Develop and consume RESTful APIs and services to enable seamless system interoperability
- Work with large-scale distributed compute frameworks to process and manage high-volume data
- Drive performance tuning, optimization, and scalability improvements
Benefits
- Vacation: 12-25 days, depending on grade
- Company paid holidays
- Personal Days
- Sick Leave
- Medical, dental, and vision coverage
- Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
- Life and disability insurance
- Employee assistance programs
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
