Staff Engineer

Graphcore•Austin, TX

9d•Hybrid

About The Position

We are looking for an experienced Staff Engineer to join our System Management team to help develop the critical interfaces used by internal and external customers to manage system state. Positioned between the hardware and customer workload, we are responsible for building the substrate upon which all other teams build upon. As part of the System Management team, you’ll collaborate across teams to make complex systems work seamlessly and reliably. While your primary focus will be on engineering excellence and system-level development, we value individuals who are versatile and hands-on — willing to contribute wherever needed, providing direct support and troubleshooting as necessary. The ideal candidate brings deep experience with HPC and cloud-native infrastructure, thrives in fast-paced, loosely scoped environments, and leads with action, decisiveness, and teamwork. The System Management team, part of the Software Platform group, helps build Graphcore products into large-scale AI solutions for our customers. The System Management Team is responsible for building the interfaces between hardware and AI software and frameworks as well as providing interfaces for public/private cloud use. This takes the form of a rack management solution that abstracts complex hardware administrative duties. As one of the first teams to work with pre-release hardware and software it’s vital you are comfortable with unproven components and capable of problem solving solutions no matter what.

Requirements

Bachelor's degree or equivalent practical experience in a relevant subject.
Experience with RESTful API development.
Experience building, deploying, and operating containerized workloads using Kubernetes and container runtimes such as Docker or Podman.
Experience with managing production Kubernetes clusters and workloads.
Programming experience with Go.
Hands-on experience deploying and operating infrastructure using Infrastructure-as-Code, source code version control, and CI/CD automation tools (e.g. Terraform/OpenTofu, Ansible, GitLab, GitHub Actions, Git version control).
Experience with Redfish for datacenter hardware management, telemetry, provisioning, and control.
Experience specifying, scoping, estimating and detailing work plans in an AGILE and SCRUM framework, including priorities, risks, issues, impacts and constraints.
Strong Linux systems engineering experience, including administration, automation, and scripting with Bash and Python.

Nice To Haves

Experience with AI coding assistants (Codex, Claude, etc).
Experience with Kubernetes operator development (Custom resources).
Experience with High Performance Computing (HPC) environments using SLURM or similar batch workload solutions.
Experience with virtualized deployments and the technologies they rely on (e.g. Open vSwitch, KVM, QEMU).
Experience with distributed object, block, and file storage (e.g., Ceph).
Experience in end-to-end deployment automation and CI of containerized services.
Complete automation of pipelines for build, test, deploy, manage, alert, destroy, rebuild.
Experience with solutions for monitoring and observability (e.g. Grafana, Prometheus, OpenSearch/ElasticSearch, Loki, Mimir, OpenTelemetry, Fluentd, Kafka).
Experience with managed switch configuration (e.g. EOS, SONiC, DNOS).
Experience with PyTorch for AI workloads.
Solid understanding of cloud and infrastructure technologies, including APIs, virtualization, networking, block storage, resource management, and monitoring systems.

Responsibilities

Ownership of software engineering efforts across the full SDLC, including implementation, automated testing, integration, and production readiness for the rack management solution.
Ownership of critical infrastructure with the need to drive issues to resolution while collaborating effectively across teams.
Configure and test new Graphcore AI hardware and systems using Continuous Deployment and Infrastructure-as-code in internal and external datacentres.
Work with our Datacenter Operations Engineers to maintain and operate the fleet of AI systems at peak performance.
Drive corrective actions for systems that are not operating correctly, working with DC operations and Graphcore Engineering as required.