Distributed Systems Engineer: Secure Sandboxes

Magic AI • San Francisco, CA

61d • Remote

About The Position

As a Software Engineer on the Supercomputing Platforms and Infrastructure team, you will build the next generation systems that power large scale AI research and deployment. You will focus on sandboxed execution environments, distributed systems orchestration, and performance optimized compute workflows. You will work closely with ML and Research teams and infrastructure teams to deliver both high throughput, scale, and strong isolation guarantees in a cluster environment.

Requirements

• Strong software engineering background (C, C++, Go, Rust, or similar systems languages).
• Experience designing or operating sandboxed or isolated execution environments (namespaces, cgroups, container runtime internals), or strong interest in this area.
• Experience building or operating distributed systems or parallel processing frameworks (scatter aggregate processing, worker pools, multi thread and multi process coordination, shared memory, atomics, merging strategies).
• Solid understanding of storage and IO subsystems (NVMe, SSD endurance, write amplification), network performance, CPU and memory resource constraints in high performance compute clusters.
• Comfortable working on low level systems (OS, threading, memory management, synchronization) as well as higher level orchestration or automation.
• Experience with cloud infrastructure (GCP, AWS, Azure, etc.) including IaC tools such as OpenTofu, Terraform, Pulumi, or CDK is a plus.
• Intellectual curiosity, strong ownership, and the ability to make tradeoffs in ambiguous environments such as latency versus throughput and isolation versus performance.

Nice To Haves

• Prior experience with GPU scheduling, RDMA networking, or bare metal HPC clusters
• Contributions to open source container runtimes or sandboxing frameworks
• Experience with kernel internals, device drivers, or SSD and NVMe endurance modeling
• Familiarity with Rust for systems programming or Go for infrastructure orchestration

Responsibilities

• Build highly scalable, highly performant, software that facilitates arbitrary code execution with strong isolation guarantees.
• Design and build systems that allow our AI models to interface with machines in various modes, interactive terminal, GUI applications, etc.
• Provision and operate high density compute and storage nodes (NVMe, high IOPS SSDs, high bandwidth networks), and build software that performs efficient load balancing, and resource utilization across them.
• Instrument and optimize end to end performance including storage IO, network bandwidth, CPU, memory, and endurance constraints.
• Develop APIs, self service platforms, and automation and tools so researchers and engineers can deploy and monitor workloads at scale.
• Troubleshoot complex infrastructure issues across OS, drivers, hardware, storage systems (local NVMe, block storage, NFS), networking, namespace isolation, and cloud or hybrid environments.
• Produce clean, documented code and developer workflows, and collaborate with SRE and security teams to ensure safe, reliable, and self serviceable compute offerings.

Benefits

• Excellent compensation and equity, generous benefits, and high impact.
• 401(k) with matching, comprehensive health, dental, and vision insurance, unlimited paid time off, visa sponsorship and relocation support
• Fast paced, mission driven environment focused on safely advancing AGI for humanity

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed
• Number of Employees: 11-50 employees