Senior Software Engineer, Crusoe Managed Data Transfer

Crusoe•San Francisco, CA

2d•$170,000 - $205,000

About The Position

Crusoe's Managed Platform Services organization builds managed services for Crusoe Cloud, including Managed Data Transfer (MDT). MDT moves petabyte-scale training data from AWS S3, and later other clouds, into Crusoe storage through a single API call. It distributes work across transfer workers, verifies every object, and requires customers to run no transfer infrastructure themselves. The Senior Software Engineer will build MDT's control plane, including its job API, scheduler, and job state management. The role will help take the service from its first commit through preview launch with design-partner customers and shape an API that can later support additional clouds and internal Crusoe transfers. The engineer will work with the Storage, Network, IAM, and Cloud Console teams and report to the Engineering Manager of Managed Data Transfer.

Requirements

At least 5 years of experience building cloud infrastructure or platform services, including taking a service from zero to general availability.
Experience designing and shipping production backend services and APIs, ideally using Go, gRPC or REST, relational databases, and Kubernetes.
Comfort with long-running, failure-prone distributed work, including idempotency, retries with backoff, leases, checkpointing, and partial failure.
Ability to design public APIs that customers can automate against and that can evolve without breaking changes.
Working knowledge of authentication, authorization, and least privilege, and care in handling customer credentials.
Experience running production services, participating in on-call, and building dashboards and alerts.
Ability to collaborate across teams with differing priorities and produce design documents that help settle decisions.
Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent professional experience.

Nice To Haves

Experience with large-scale data movement, such as S3 multipart uploads, rclone, s5cmd, AWS DataSync, or Google Storage Transfer Service.
Hands-on experience with cross-cloud identity, such as AWS IAM and STS, GCP service-account impersonation, or Azure workload identity.
Experience with storage systems such as object storage, NFS, or parallel file systems, or with high-throughput networking.
Experience with job orchestration systems such as Temporal, Argo, or Kubernetes operators.
Experience building on GPU or AI infrastructure.

Responsibilities

Design and build the data-transfer API for creating, listing, retrieving, and canceling jobs, using a provider-neutral model for sources and destinations.
Build the scheduler and state machine to divide bucket listings into units of work, assign them to regional worker pools, and track progress, retries, and resumption after failures.
Make large-scale jobs safe to retry and cancel, implement end-to-end checksum verification, and provide clear errors for permission or destination problems.
Implement secure access to customer buckets using short-lived, least-privilege credentials, beginning with AWS role assumption and integrating with Crusoe IAM and audit logging.
Ship the CLI and collaborate with partner teams on Terraform and Console experiences for creating and monitoring transfers.
Instrument the service with metrics, logs, and alerts; write runbooks; and participate in the team's on-call rotation.
Agree on interfaces with Storage, Network, IAM, and Console teams, and work with Product and Solutions Architects on early customer onboarding.