About The Position

Nscale is seeking a Principal Site Reliability Engineer, Infrastructure Architecture, to serve as an architectural authority for the design, connectivity, and recovery of its AI and HPC infrastructure. This design-first role shares ownership of the reference architecture and operating model with an existing architecture lead. The position works alongside teams responsible for platform bring-ups, upgrades, migrations, and operations, and participates in major incidents when architectural issues are involved. The role owns end-to-end reference architecture across compute, network fabric, storage, control plane, and core services. It defines multi-site and multi-region topologies, failure domains, tenant isolation, trust boundaries, target-state architecture, and a multi-year roadmap. The role also establishes architectural standards, leads cross-functional reviews, co-owns operational practices with SRE, and ensures teams can deploy, upgrade, and recover their systems independently. The position includes planning site bring-ups and fleet changes, running disaster recovery tests and game days, using incident and on-call data to guide redesign, and mentoring senior engineers. The role may be eligible for bonus, equity, and/or commission programs in addition to base salary.

Requirements

  • 10 or more years building and operating large-scale production infrastructure, including on-call for systems the candidate designed.
  • Several years of experience as a staff-, principal-, or architect-level engineer.
  • A track record of architectures designed and running in production, with the ability to explain tradeoffs, failures, and potential improvements.
  • Experience leading major incidents and running large migrations or site bring-ups.
  • Working knowledge of change management, staged rollouts, SLO-based alerting, and post-incident reviews.
  • Deep expertise in distributed systems design, including consistency models, consensus and replication, partition handling, and graceful degradation.
  • Strong networking fundamentals at the design level, including L3 fabrics, BGP, EVPN/VXLAN, segmentation, and multi-site connectivity.
  • Experience designing secure multi-tenant platforms, including identity, PKI and secrets management, network isolation, and zero-trust boundaries.
  • Clear technical writing skills, with design documents, RFCs, and ADRs that engineers can use to build systems.
  • Ability to drive architectural decisions across teams without direct authority, including rejecting designs that are not robust.
  • Strong software engineering skills, including prototyping designs, reviewing production code, and identifying implementation drift from the intended architecture.
  • Sound judgment when reliability, velocity, and cost involve tradeoffs, and ability to explain those tradeoffs to engineers and executives.

Nice To Haves

  • Experience architecting AI or HPC platforms at scale, including GPU clusters, InfiniBand or RoCE fabrics, and workload schedulers such as Slurm or Kubernetes-based GPU scheduling.
  • Experience designing sovereign, regulated, or air-gapped cloud environments.
  • Bare-metal and hybrid cloud architecture experience, including provisioning pipelines, CMDB/DCIM as a source of truth, and hardware lifecycle automation.
  • Observability architecture experience in high-cardinality, high-throughput environments.
  • Kubernetes platform architecture experience across many clusters and sites.
  • Experience with structured design analysis methods such as FMEA, threat modelling, or C4 diagrams.
  • A history of significant reliability or scalability gains achieved through redesign rather than patching.

Responsibilities

  • Own the end-to-end reference architecture for AI and HPC infrastructure, including compute, network fabric, storage, control plane, and core services.
  • Design multi-site and multi-region topologies, including failure domains, tenant isolation, and secure service enclaves.
  • Design separation between the control plane and data plane so management-layer outages do not take down customer workloads.
  • Define target-state architecture and a multi-year roadmap, including systems to retire, consolidate, or rebuild.
  • Lead architecture reviews across Engineering, Network Operations, and Fleet Operations; write and approve RFCs and architecture decision records.
  • Set architectural standards for SLOs, observability, capacity models, dependency maps, and documented failure modes.
  • Run pre-mortems and red-team reviews to identify single points of failure and hidden coupling.
  • Join major incidents with architectural root causes and follow fixes through to design changes.
  • Co-own the operating model with SRE, including operational readiness reviews, incident severity and post-incident review standards, and production access and break-glass policies.
  • Ensure teams can deploy, upgrade, and recover their systems without requiring the role holder to be present.
  • Plan and review site bring-ups, fleet upgrades, and migrations of live customer workloads.
  • Run disaster recovery tests and game days.
  • Use incident and on-call data to decide what to redesign.
  • Mentor senior engineers in systems design and support distributed architectural knowledge and decision-making.

Benefits

  • Highly competitive base salary and equity package, with reviews every 12 months.
  • Flexible workplace and autonomy to shape the workday.
  • Remote-first team.
  • Medical, dental, and vision benefits.
  • Flexible paid time off.
  • Parental leave.
  • Retirement plan participation.
  • The role may be eligible for bonus, equity, and/or commission programs.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service