Staff Software Engineer, Cloud Monitoring Service (Distributed Systems)

Crusoe•San Francisco, CA

2d•$215,000 - $260,000•Remote

About The Position

We are seeking a Staff Software Engineer to lead distributed systems design and development within Crusoe Cloud's Cloud Monitoring Service. This team builds the observability backbone of Crusoe Cloud: the telemetry systems that collect, process, store, and serve the metrics and logs our customers depend on to run AI workloads at scale. You will own the design and evolution of high-throughput, multi-tenant data systems, from collection at the edge through ingestion, storage, and query. This is a full-time position.

Requirements

Deep, hands-on experience designing and operating distributed systems at scale. You have solved real problems in sharding, replication, consistency, load balancing, and concurrency, not just studied them.
Experience building or operating large-scale data infrastructure such as time-series databases, log aggregation, streaming pipelines, or distributed tracing backends. Familiarity with technologies like Prometheus, VictoriaMetrics, Loki, OpenTelemetry, Kafka, Vector, or similar.
Strong programming fundamentals in Go or another modern compiled language (Go strongly preferred). Comfort with Kubernetes, microservices, and CI/CD as the operating environment for everything you build.
You proactively scope ambiguous problems, surface non-functional requirements without being prompted, and reason about tradeoffs in terms of customer and business outcomes, not just technical elegance.
On-call experience on a customer-facing service. You treat incidents as design feedback and systematically eliminate the conditions that cause them.
A track record of driving technical outcomes that span team boundaries, and the communication skills to bring product, support, and adjacent engineering teams along.
You make the engineers around you better through design guidance, thoughtful review, and honest, constructive feedback.
8+ years of software development experience, with sustained ownership of production distributed systems.

Responsibilities

Own the architecture and evolution of large-scale telemetry pipelines, including high-volume ingestion, stream processing, time-series and log storage, and low-latency query paths. Design systems that stay correct, available, and cost-efficient as data volume grows 10x.
Design services that are highly scalable, durable, and fair across tenants. Solve the hard problems in this space: hot shards, high-cardinality data, noisy neighbors, backpressure, retention and compaction at scale, and graceful degradation under load.
Build for operability from day one. Improve pipeline reliability and data freshness, reduce on-call burden through better system design rather than more process, and participate in a customer-facing on-call rotation, leading by example.
Set the technical direction for the team's distributed systems work. Drive design reviews, identify one-way door decisions early, and raise the bar on how the team scopes, builds, and operates systems.
Work with product, compute, networking, and platform teams to make sure observability decisions are made with full context. Represent the team's technical position in cross-org conversations.
Coach senior and mid-level engineers through design work, code review, and incident response. Build patterns and frameworks that make the team better without requiring your direct involvement.