Staff Software Engineer, Observability/Site Reliability

PsiQuantum•Palo Alto, CA
•$164,900 - $276,800

About The Position

PsiQuantum is seeking a Staff Software Engineer to join its Observability team, which builds and operates the telemetry platform used to monitor quantum-computing systems and their supporting infrastructure. The platform collects metrics, logs, and traces from on-premises Kubernetes clusters, cloud infrastructure, and custom quantum-computer hardware, including control electronics and cryostats. Engineering, operations, hardware, and research teams use this data to operate systems, identify problems, and investigate failures. The role focuses on OpenTelemetry instrumentation and collection, operating and scaling Prometheus and Grafana, and creating dependable dashboards and alerts. It includes close collaboration with hardware teams to determine what devices should report and to reliably deliver device data to the platform. The engineer will own major parts of the observability stack from design through operation, participate in the on-call rotation, and establish instrumentation conventions for other teams. The position combines hands-on engineering and technical leadership, with an emphasis on sound judgment and simple, maintainable systems.

Requirements

  • At least 8 years of professional software, infrastructure, or site reliability engineering experience, including significant experience building and operating production observability systems.
  • A Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or a closely related technical discipline.
  • Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alertmanager, exporters, dashboard design, and operating both reliably at scale.
  • Production experience with OpenTelemetry, including the Collector, OTLP, SDKs and automatic instrumentation, and semantic conventions.
  • Hands-on experience running workloads on self-managed, on-premises Kubernetes and in the cloud, with containers, Helm, and infrastructure as code such as Terraform.
  • Strong programming skills in Python and at least one of Go or Rust, including the ability to write production-quality exporters, Collector components, and automation.
  • Solid knowledge of Linux and networking fundamentals, including processes, filesystems, resource limits, DNS, TCP/IP, and TLS, and experience debugging distributed systems from the host up.
  • Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing actionable alerts.
  • Staff-level technical leadership, including leading design work, setting standards adopted by other teams, mentoring engineers, influencing across teams, and communicating tradeoffs to technical and non-technical stakeholders.
  • Proficiency with modern AI/LLM engineering tools, such as coding copilots and agents, model APIs, and retrieval workflows, and a disciplined approach to privacy, security, hallucination risk, and generated-code review.
  • Ability to work closely with hardware, electronics, and research teams across sites and time zones in a fast-moving environment with evolving requirements.

Nice To Haves

  • Practical experience with GitLab and GitLab CI.
  • Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, VictoriaMetrics, Loki, Tempo, OpenSearch, or ClickHouse.
  • Experience collecting telemetry from laboratory or industrial equipment over Modbus, OPC UA, SNMP, SCPI, serial interfaces, or vendor SDKs.
  • Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing.
  • Experience with high-frequency or high-cardinality time-series data, including downsampling, aggregation, and data lifecycle management.
  • Experience running observability across multiple sites and hybrid on-premises and cloud environments, including sites with constrained network connectivity.
  • Experience with eBPF-based observability, continuous profiling, or Kubernetes operators.
  • Experience with SLO tooling, synthetic monitoring, or incident management platforms such as PagerDuty.
  • Familiarity with VAST Data, Kafka, and AWS.
  • Contributions to OpenTelemetry, Prometheus, Grafana, or related open-source projects.
  • A background in physics or quantum computing; curiosity, systems thinking, and motivation to learn the domain are also valued.

Responsibilities

  • Design, deploy, and operate the observability platform across on-premises Kubernetes and cloud, including collection, storage, querying, visualization, and alerting for metrics, logs, and traces.
  • Roll out OpenTelemetry across the fleet by deploying and configuring Collectors, designing telemetry pipelines, and helping service teams adopt OTel SDKs and instrumentation in Python, Rust, and other languages.
  • Run Prometheus in production, including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long-term storage.
  • Manage Grafana dashboards, alerts, and data sources as code so they are versioned, reviewed, and reproducible across environments.
  • Work with hardware teams to instrument custom devices, control electronics, cryostats, and lab equipment; build exporters and collectors to bring device telemetry into the platform.
  • Define and publish telemetry standards for metric and label naming, cardinality budgets, structured logging, trace context propagation, and resource attributes.
  • Partner with service owners and operations teams to define SLOs and alerts that reflect real impact, reduce alert noise, and make pages actionable.
  • Participate in the team's on-call rotation, respond to incidents affecting the observability platform, help teams diagnose production issues with telemetry, and lead blameless post-incident reviews through to completed fixes.
  • Operate the platform as a production service, including capacity planning, cost management, upgrades, backup and recovery, and monitoring the monitoring stack, using GitLab CI/CD and infrastructure as code.
  • Use AI and LLM tools to accelerate engineering and operational work, including coding assistance, query and dashboard generation, and incident analysis, while applying review, security, and validation practices.
  • Work with the team's technical lead on architecture and roadmap; lead design reviews, mentor engineers, document consequential decisions, and coordinate across software, hardware, research, infrastructure, and operations teams.
  • Configure OTel Collectors and Prometheus scrape targets, write exporters, and help teams instrument new services and devices.
  • Write code and configuration, review merge requests for platform and dashboard changes, and simplify difficult-to-operate pipelines.
  • Work with hardware teams to understand devices and cryostats, agree on measurements and alerts, and integrate their data.
  • Monitor platform health and alert quality, handle on-call pages, and help teams debug using metrics, logs, and traces.
  • Work through storage, retention, cost, and tooling tradeoffs with the technical lead; unblock engineers and align near-term work with the roadmap.
  • Use LLM-assisted tools for queries, configuration, code, and documentation, verifying outputs to production-infrastructure standards.

Benefits

  • Competitive health coverage for the employee and dependents.
  • 401(k) with company match.
  • Equity grants.
  • Access to financial wellness tools and planning resources.
  • Wellness benefits.
  • Family support programs.
  • Life and disability insurance.
  • Paid leave programs.
  • Company-designated paid holidays.
  • Discretionary time off (DTO).
  • End-of-year company shutdown.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service