Principal Engineer

Graphcore•Milpitas, CA

13d

About The Position

We are looking for an experienced Principal Engineer to join our System Management team and help lead the development of critical interfaces used by internal and external customers to manage system state. You will provide technical leadership within assigned areas of System Management, guide architecture and implementation choices, mentor engineers and translate broader technical direction into effective execution. This is a hands-on engineering role for someone who can lead complex technical work, improve reliability and operational readiness, and collaborate effectively across multiple engineering disciplines. The System Management team sits within the Software Platform group and helps build Graphcore products into large-scale AI solutions for our customers. The team is responsible for developing the interfaces between hardware, AI software and frameworks, as well as providing interfaces for public and private cloud environments. This includes system management capabilities that abstract complex hardware administration and enable reliable deployment and operation at scale. As one of the first teams to work with new hardware and software, we regularly solve complex system-level problems in environments where components and interfaces are still evolving. The role requires strong technical judgement, adaptability and an ability to work effectively across engineering teams.

Requirements

Bachelor’s degree or equivalent practical experience in a relevant subject.
Substantial experience designing, building and operating Linux-based infrastructure or distributed systems.
Demonstrated experience providing technical leadership for complex engineering initiatives involving multiple teams or stakeholder groups.
Experience influencing architecture and technical decisions across team boundaries without relying on formal authority.
Experience translating broad technical goals into scoped plans, milestones, technical decisions, risks and delivery priorities.
Strong experience developing RESTful APIs and programming in Go, with Bash and Python used for systems automation.
Deep practical experience with Kubernetes, container runtimes and operating production workloads.
Hands-on experience with Infrastructure-as-Code, source control and CI/CD technologies such as Terraform/OpenTofu, Ansible, GitLab, GitHub Actions and Git.
Experience with hardware-management interfaces such as Redfish, IPMI or equivalent management systems.
Strong Linux systems engineering, troubleshooting and operational debugging capability.
Demonstrated ability to develop other engineers through technical mentoring, design reviews and coaching.
Clear communication skills, with the ability to persuade, build alignment and bring stakeholders together around practical technical outcomes.

Nice To Haves

Experience using AI coding assistants effectively within professional engineering workflows.
Experience developing Kubernetes operators and custom resources.
Experience with High Performance Computing environments using SLURM, LSF or similar workload-management systems.
Experience with virtualisation technologies such as Open vSwitch, KVM and QEMU.
Experience with distributed object, block and file storage technologies such as Ceph.
Experience with monitoring and observability platforms such as Grafana, Prometheus, OpenSearch/Elasticsearch, Loki, Mimir or OpenTelemetry.
Experience configuring managed network switches using technologies such as EOS, SONiC or DNOS.
Experience supporting AI infrastructure or PyTorch workloads.

Responsibilities

Convert agreed System Management direction into technical plans, engineering priorities and deliverable work for assigned areas.
Provide technical leadership for architecture and design decisions, building alignment across collaborating teams and documenting important technical trade-offs.
Act as a technical authority for assigned areas of System Management, leading the delivery of large and complex engineering initiatives and coordinating technical plans, dependencies, risks and decisions.
Provide technical direction and mentoring to engineers working across system management, hardware lifecycle management, deployment automation and production operations.
Take responsibility for key technical outcomes across the full software lifecycle, including design, implementation, automated testing, integration, deployment, observability and production readiness.
Identify systemic reliability, scalability and operability issues and lead practical improvements across the platform.
Collaborate with Hardware, Firmware, Platform Software and Datacenter Operations teams to diagnose system-level issues and improve end-to-end product behaviour.
Improve engineering standards and working practices, including CI/CD, Infrastructure-as-Code, automated testing, release safety and learning from operational incidents.
Act as a senior technical escalation point for complex issues while creating reusable knowledge, tooling and automation that reduce future operational effort.