Senior Systems Software Engineer - Accelerated Spark Applications

NVIDIA•Champaign, IL

1d•$184,000 - $287,500

About The Position

NVIDIA is seeking a Senior Systems Software Engineer for its Apache Spark Acceleration group. The group accelerates Apache Spark applications on GPUs through an open-source plugin and cuDF Spark library, which is integrated with several cloud and data platforms. This role supports an internal hosted Spark service that runs production data pipelines for NVIDIA’s data platform at thousands of jobs per hour on Kubernetes and AWS under a customer SLA. The service packages the cuDF Spark plugin and a CPU/GPU workload router as an API-based service built on Spark Connect, Kyuubi, Helm, and Flux. The engineer will work across the service, GPU Spark runtime, and workload router.

Requirements

BS, MS, or PhD in Computer Science, Computer Engineering, or a closely related field, or equivalent experience.
9+ years of experience in software development.
Strong distributed-systems fundamentals and proficiency in Java or Scala, plus C++ or Python.
Hands-on experience operating services on Kubernetes, on-premise or in the cloud.
Experience owning or reviewing a long-lived server, such as REST front ends, proxies or request routers, or JDBC or Thrift services.
Experience supporting enterprise or internal production customers, including on-call.
Experience using AI tools in software development.

Nice To Haves

Experience with Apache Spark internals, Spark Connect, or Kyuubi.
Delta Lake, Apache Iceberg, or Unity Catalog integration work.
GPU data processing with cuDF or CUDA experience.
Significant contributions to collaborative, shared-development data platform projects such as Apache Spark, Velox, Presto, Apache Arrow, or Apache DataFusion.

Responsibilities

Design, build, and operate a hosted Spark service control plane on Kubernetes, including the Java REST API, Spark Connect runner, Kyuubi, and deployment using Helm charts and Flux.
Own code review for one or more services end to end and keep them releasable on the release train.
Extend the cuDF Spark runtime and CPU/GPU workload router in Scala and Java, including interoperability with Delta Lake, Apache Iceberg, and Unity Catalog.
Investigate production failures on clusters running more than a thousand concurrent jobs with customer and upstream library teams, and drive fixes to closure.
Instrument metrics, dashboards, and alerts; participate in the on-call rotation; and keep runbooks current.
Provide recommendations and feedback to teams on infrastructure, continuous integration, and testing strategy.