Software Engineer - Inference Product

Baseten•San Francisco, CA
•$200,000 - $400,000

About The Position

Baseten is seeking a product-minded backend and distributed systems engineer to build the API layer powering Model APIs, its hosted endpoints for open-source models. The team owns the systems every Model API request passes through, including authentication, rate limiting, routing, metering, and billing. The role focuses on making these systems fast, reliable, and easy for developers to use, while shipping products that influence how developers use models at scale. This role sits at the intersection of infrastructure and product. The systems built determine customer performance, how capacity is shared across a global GPU fleet, and how inference is packaged and sold. The engineer will work on a small, high-impact team alongside inference and product teams, owning work from design through production. Example initiatives include Baseten Grid, Baseten Hosted Tools, and Frontier Gateway.

Requirements

  • Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field, or equivalent practical experience.
  • At least 3 years building and operating distributed systems, backend infrastructure, or large-scale APIs where reliability, latency, and scale are first-class concerns.
  • Proven track record of owning low-latency, reliable backend services, including rate limiting, authentication, quotas, metering, and migrations.
  • Infrastructure instincts and performance experience, including profiling, tracing, capacity planning, and SLO management.
  • Ability to debug performance and reliability issues across multiple stack layers, from application behavior to runtime and infrastructure internals.
  • Strong sense of developer experience and consideration of how systems are used, not just how they work.
  • Eagerness to learn new languages, frameworks, and systems, and genuine interest in inference engineering.
  • Excellent written communication and collaboration skills, including writing clear design documents and working across functions.

Nice To Haves

  • Deep Kubernetes experience, including operators and custom resources, plus familiarity with service meshes or API gateways.
  • Experience with distributed scheduling, autoscaling, or service orchestration.
  • Background in developer-facing infrastructure or APIs, or contributions to open-source infrastructure or ML systems.
  • Familiarity with observability tooling, CI/CD systems, or release automation.
  • Experience with model serving or inference systems.

Responsibilities

  • Build and operate the API layer behind Model APIs, including request handling, authentication, rate limiting, and routing, with high availability and low added latency.
  • Own usage metering and billing infrastructure, providing accurate real-time accounting for requests and supporting new pricing and packaging models.
  • Build rate-limiting and capacity systems that balance customer throughput and fleet utilization at scale and in real time.
  • Ship API-layer products that change how developers use models, including model routing and features that make Model APIs easier to build on.
  • Turn platform capabilities into a strong product experience through usage visibility, self-serve controls, and clear, predictable API behavior.
  • Instrument the edge with observability and alerting so issues are caught and fixed before customers notice.
  • Partner with Model APIs, inference, and product teams to deliver new capabilities to customers quickly and safely.
  • Own projects end to end, from design through deployment, on-call, and iteration based on customer feedback.

Benefits

  • Competitive compensation, including meaningful equity.
  • For U.S. employees: 100% coverage of medical, dental, and vision insurance for employees and dependents.
  • Flexible PTO policy, including a company-wide Winter Break when offices are closed from Christmas Eve through New Year's Day.
  • Paid parental leave.
  • Fertility and family-building stipend through Carrot.
  • For U.S. employees: company-facilitated 401(k).
  • Exposure to a variety of ML startups, offering learning and networking opportunities.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service