Staff Engineer

Graphcore • Milpitas, CA

9d

About The Position

Graphcore is seeking an experienced Staff Engineer to join their System Management team. This role involves developing critical interfaces for managing system state, acting as a bridge between hardware and customer workloads. The team is responsible for building the foundation upon which other teams operate. The Staff Engineer will collaborate across teams to ensure seamless and reliable system operation. While the primary focus is on engineering excellence and system-level development, versatility and hands-on problem-solving are highly valued. The ideal candidate will have deep experience with HPC and cloud-native infrastructure, be comfortable in fast-paced, loosely scoped environments, and demonstrate leadership through action, decisiveness, and teamwork. The System Management team, part of the Software Platform group, contributes to building Graphcore products into large-scale AI solutions by developing interfaces between hardware and AI software/frameworks, as well as interfaces for public/private cloud use. This includes a rack management solution that abstracts complex hardware administrative duties. As an early team working with pre-release hardware and software, comfort with unproven components and strong problem-solving skills are essential.

Requirements

• Bachelor's degree or equivalent practical experience in a relevant subject.
• Experience with RESTful API development.
• Experience building, deploying, and operating containerized workloads using Kubernetes and container runtimes such as Docker or Podman.
• Experience with managing production Kubernetes clusters and workloads.
• Programming experience with Go.
• Hands-on experience deploying and operating infrastructure using Infrastructure-as-Code, source code version control, and CI/CD automation tools (e.g. Terraform/OpenTofu, Ansible, GitLab, GitHub Actions, Git version control).
• Experience with Redfish for datacenter hardware management, telemetry, provisioning, and control.
• Experience specifying, scoping, estimating and detailing work plans in an AGILE and SCRUM framework, including priorities, risks, issues, impacts and constraints.
• Strong Linux systems engineering experience, including administration, automation, and scripting with Bash and Python.

Nice To Haves

• Experience with AI coding assistants (Codex, Claude, etc).
• Experience with Kubernetes operator development (Custom resources).
• Experience with High Performance Computing (HPC) environments using SLURM or similar batch workload solutions.
• Experience with virtualized deployments and the technologies they rely on (e.g. Open vSwitch, KVM, QEMU).
• Experience with distributed object, block, and file storage (e.g., Ceph).
• Experience in end-to-end deployment automation and CI of containerized services. Complete automation of pipelines for build, test, deploy, manage, alert, destroy, rebuild.
• Experience with solutions for monitoring and observability (e.g. Grafana, Prometheus, OpenSearch/ElasticSearch, Loki, Mimir, OpenTelemetry, Fluentd, Kafka).
• Experience with managed switch configuration (e.g. EOS, SONiC, DNOS).
• Experience with PyTorch for AI workloads.
• Solid understanding of cloud and infrastructure technologies, including APIs, virtualization, networking, block storage, resource management, and monitoring systems.

Responsibilities

• Ownership of software engineering efforts across the full SDLC, including implementation, automated testing, integration, and production readiness for the rack management solution.
• Ownership of critical infrastructure with the need to drive issues to resolution while collaborating effectively across teams.
• Configure and test new Graphcore AI hardware and systems using Continuous Deployment and Infrastructure-as-code in internal and external datacentres.
• Work with our Datacenter Operations Engineers to maintain and operate the fleet of AI systems at peak performance.
• Drive corrective actions for systems that are not operating correctly, working with DC operations and Graphcore Engineering as required.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Associate degree