Software Engineer, Observability

SpaceXSI • Memphis, TN

8h

About The Position

SpaceXSI is building Super Intelligence to help humanity understand the universe. Its teams develop the Super Intelligence stack, including supercomputers, models, and applications. The Software Engineering team builds software systems and platforms that keep datacenters operating reliably at the scale needed for frontier AI training and inference. The team makes large, heterogeneous, always-on clusters observable and supports automated operations, faster node remediation, and increased GPU uptime. It works closely with datacenter operations, AI research, and facility infrastructure teams to turn operational data into clear insight and action and to support site-wide reliability.

The Software Engineer, Observability designs and operates production systems for telemetry ingestion, dashboards, monitoring, alerting, operational workflows, and automation. The role works directly with users in person to incorporate feedback and improve processes, and may require extended hours or weekends for mission-critical deadlines and participation in a weeklong on-call rotation.

Requirements

• Bachelor's degree in computer science, engineering, mathematics, or a STEM discipline and at least 3 years of professional software development experience, or at least 5 years of professional software development experience in lieu of a degree.
• Programming experience in Python, Rust, C++, Java, or similar languages.
• Experience with Postgres, MySQL, MongoDB, DynamoDB, or similar database management systems.
• Demonstrated ownership of correctness-sensitive systems, such as state machines, workflows, or operational data where inaccurate state has real-world impact.
• Experience collaborating with cross-functional teams.
• Ability to work extended hours or weekends as needed for mission-critical deadlines.
• Ability to participate in a weeklong rotational on-call schedule.

Nice To Haves

• Experience building or operating large-scale observability, telemetry, or real-time analytics systems.
• Experience with time-series or analytical databases such as Prometheus/VictoriaMetrics, ClickHouse, or InfluxDB, and streaming or high-volume telemetry pipelines.
• Experience building observability dashboards and alerting with Grafana, New Relic, Splunk, or similar tools.
• Frontend experience with React, Angular, or a comparable JavaScript framework, building intuitive user interfaces for critical operations.
• Experience designing and shipping multi-service systems for mission-critical applications, including RESTful APIs and data stores, plus at least one of user interfaces, pipelines, or authentication.
• Experience operating services on cloud or on-premises infrastructure with Docker/Kubernetes in production and automated CI/CD deployments such as ArgoCD or GitHub Actions.
• On-call discipline and a track record of owning internal platforms end to end with production rigor in rapidly changing environments.
• Experience with HPC or supercomputing environments, such as schedulers including Slurm, GPU/accelerator telemetry, high-speed fabrics, or parallel filesystems.
• Experience integrating industrial/OT data sources, such as SCADA platforms including Ignition, tag historians, Modbus/OPC-UA, or power/cooling infrastructure telemetry.
• Master's degree in Computer Science or a related field.

Responsibilities

• Design and implement scalable, fault-tolerant real-time data ingestion pipelines for high-volume telemetry from compute nodes, GPUs and accelerators, schedulers, storage, network fabric, and facility systems.
• Develop frontend visualizations, interactive dashboards, and UI components that present processed data in real time to operators, capacity planners, and engineering stakeholders.
• Design, build, and operate multi-service production systems—including user interfaces, APIs, data pipelines, authentication, and on-call operations—for reliability and uptime monitoring, predictive alerting, upgrade and change risk management, and site-wide impact assessments.
• Build and maintain operational software stacks, including repair trackers, vendor RMA workflows, operational infrastructure, and integrations with ticketing, inventory and rack systems, telemetry stores, and vendor portals.
• Drive automation to reduce manual cluster-monitoring effort and improve anomaly-detection accuracy.
• Maintain the reliability of the tools, including uptime, data integrity, reconciliation, access control, and safe deployments.
• Work in person with users to incorporate feedback and improve processes that increase GPU uptime.
• Investigate operational problems, find efficient solutions, and apply first-principles thinking.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level