Staff Software Engineer, Observability/Site Reliability

PsiQuantum•Palo Alto, CA

17h

About The Position

PsiQuantum is hiring a Staff Software Engineer to join its Observability team, which builds and operates the telemetry platform used to understand the behavior of its quantum computing systems and supporting infrastructure. The platform collects metrics, logs, and traces from on-premises Kubernetes clusters, cloud infrastructure, and specialized hardware such as control electronics and cryostats. Engineering, operations, hardware, and research teams use this information to operate systems, identify problems early, and investigate failures. The role focuses on OpenTelemetry instrumentation and collection, operating Prometheus and Grafana, and creating useful dashboards and actionable alerts. A substantial part of the work involves collaborating with hardware teams to determine what devices should report and reliably move telemetry from devices into the platform. The engineer will own major areas of the observability stack from design through operations, participate in the on-call rotation, and establish instrumentation conventions for other teams. The position combines hands-on engineering with Staff-level technical leadership, sound judgment in ambiguous situations, and an emphasis on simple, maintainable systems. PsiQuantum is developing utility-scale, fault-tolerant quantum computers using silicon photonics, with government-funded projects supporting facilities in Brisbane, Australia, and Chicago, Illinois. Its software and industry teams also work with major companies on applications for quantum computing.

Requirements

At least 8 years of professional software, infrastructure, or site reliability engineering experience, including significant experience building and operating production observability systems.
A Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or a closely related technical discipline.
Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alertmanager, exporters, dashboard design, and reliable operation at scale.
Production experience with OpenTelemetry, including the Collector, OTLP, SDKs, automatic instrumentation, and semantic conventions.
Hands-on experience running workloads on self-managed, on-premises Kubernetes and in the cloud, using containers, Helm, and infrastructure as code such as Terraform.
Strong programming skills in Python and at least one of Go or Rust, including the ability to write production-quality exporters, Collector components, and automation.
Solid knowledge of Linux and networking fundamentals, including processes, filesystems, resource limits, DNS, TCP/IP, and TLS, and experience debugging distributed systems from the host up.
Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing alerts that people act on.
Staff-level technical leadership experience, including leading design work, setting standards adopted by other teams, mentoring engineers, influencing across teams, and communicating tradeoffs to technical and non-technical stakeholders.
Proficiency with modern AI/LLM engineering tools, such as coding copilots and agents, model APIs, and retrieval workflows, with disciplined attention to privacy, security, hallucination risk, and generated-code review.
Ability to work closely with hardware, electronics, and research teams across sites and time zones in a fast-moving environment with evolving requirements.

Nice To Haves

Practical experience with GitLab and GitLab CI.
Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, VictoriaMetrics, Loki, Tempo, OpenSearch, or ClickHouse.
Experience collecting telemetry from laboratory or industrial equipment using protocols such as Modbus, OPC UA, SNMP, SCPI, or serial interfaces, or through vendor SDKs.
Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing.
Experience with high-frequency or high-cardinality time-series data, including downsampling, aggregation, and data lifecycle management.
Experience running observability across multiple sites and hybrid on-premises and cloud environments, including sites with constrained network connectivity.
Experience with eBPF-based observability, continuous profiling, or Kubernetes operators.
Experience with SLO tooling, synthetic monitoring, or incident management platforms such as PagerDuty.
Familiarity with VAST Data, Kafka, and AWS.
Contributions to OpenTelemetry, Prometheus, Grafana, or related open-source projects.
A background in physics or quantum computing; it is helpful but not required.

Responsibilities

Design, deploy, and operate the observability platform across on-premises Kubernetes and cloud, including collection, storage, querying, visualization, and alerting for metrics, logs, and traces.
Roll out OpenTelemetry across the fleet by deploying and configuring Collectors, designing telemetry pipelines, and helping service teams adopt OTel SDKs and instrumentation in Python, Rust, and other languages.
Operate Prometheus in production, including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long-term storage.
Manage Grafana dashboards, alerts, and data sources as code so they are versioned, reviewed, and reproducible across environments.
Collaborate with hardware teams to instrument custom devices, control electronics, cryostats, and other lab equipment, and build exporters and collectors that bring device telemetry into the platform.
Define and publish telemetry standards for metric and label naming, cardinality budgets, structured logging, trace context propagation, and resource attributes.
Partner with service owners and operations teams to define SLOs and alerts that reflect real impact, reduce alert noise, and make pages actionable.
Participate in the on-call rotation, respond to observability platform incidents, help teams diagnose production issues using telemetry, and lead blameless post-incident reviews through to completed fixes.
Operate the platform as a production service, including capacity planning, cost management, upgrades, backup and recovery, and monitoring of the monitoring stack, using GitLab CI/CD and infrastructure as code.
Use AI and LLM tooling to accelerate engineering and operational work, including coding assistance, query and dashboard generation, and incident analysis, while applying review, security, and validation practices.
Work with the team's technical lead on architecture and roadmap; lead design reviews, mentor engineers, document consequential decisions, and coordinate with software, hardware, research, infrastructure, and operations teams.
Configure OTel Collectors and Prometheus scrape targets, write exporters, and help teams instrument new services and devices.
Write code and configuration, review merge requests for platform and dashboard changes, and simplify difficult-to-understand or operate pipelines.
Learn how new devices or cryostats behave, agree with hardware teams on what to measure and alert on, and establish telemetry flow.
Monitor platform health and alert quality, handle on-call pages, and help teams debug with metrics, logs, and traces.
Work through storage, retention, cost, and tooling tradeoffs with the technical lead, unblock other engineers, and align near-term work with the roadmap.
Use LLM-assisted tools for queries, configuration, code, and documentation, verifying their output to production-infrastructure standards.