Software Engineer, Data Ingestion

Wayve • Sunnyvale, CA

1d • Hybrid

About The Position

The Software Engineer, Data Ingestion will work on Wayve’s Ingestion Systems team within the AI Platform organisation. The team builds the data foundations behind Wayve’s self-driving technology, processing large volumes of real-world driving data for annotation, model training and evaluation. Its systems operate at petabyte scale, with an emphasis on reliable, efficient and operable data flows.

The role focuses on operating and improving ingestion pipelines: diagnosing failures and problematic data, optimising Spark-based processing, making workflows more resilient, and improving monitoring and recovery. The engineer will also help prioritise datasets needed by downstream teams and contribute to the technical direction of Wayve’s ingestion platform. The work includes varied data formats from partners, suppliers and other third-party sources, as well as orchestration across dependencies, queues, retries and priority flows.

Wayve operates a hybrid working model, combining in-person collaboration in dedicated offices with remote work. The company describes itself as a fast-moving environment where systems and ways of working are still being established, and where employees are expected to take ownership and contribute to building them.

Requirements

• Strong production experience with Apache Spark and Python.
• Experience building, operating or debugging large-scale ingestion, ETL or distributed data-processing pipelines.
• Comfort diagnosing production failures involving corrupt data, stalled queues, retries and complex dependencies.
• Understanding of optimising high-throughput systems for reliability, performance and cost.
• Ability to learn an existing system quickly, work independently and deliver practical improvements in a fast-moving environment.

Nice To Haves

• Experience with Airflow, Flyte, Databricks, Delta Lake, Scala or Java.

Responsibilities

• Debug and unblock failing ingestion pipelines, stalled jobs and processing queues.
• Investigate corrupt, malformed or unexpected data from a variety of sources.
• Optimise Spark jobs and batch-processing pipelines for throughput, reliability and compute efficiency.
• Design and implement resilient pipelines that prevent individual bad data segments from blocking wider workflows.
• Improve monitoring, retries and failure recovery across multi-step ingestion workflows.
• Help prioritise and unblock important datasets for downstream annotation, data science and model-training teams.
• Contribute to the technical direction of Wayve’s ingestion platform.
• Improve the performance, cost and operability of the Spark-based data platform.
• Work on orchestration across complex dependencies, queues, retries and priority data flows, and ingest varied data formats from partners, suppliers and other third-party sources.

Benefits

• Salaries benchmarked against the market annually.
• Meaningful equity.
• Relocation support and visa sponsorship where applicable.
• Hybrid working and core hours.
• Learning and development budgets, with support for training, conferences and growth.
• Comprehensive benefits including health insurance, dental insurance, enhanced maternity and paternity leave, retirement or pension where applicable, access to therapists, wellbeing partnerships and team socials.
• Opportunity to work hands-on in vehicle workshops and labs.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed