Senior Software Engineer, AI Gateway

Docker

1d • $160,900 - $260,700 • Remote

About The Position

Docker is hiring a Senior Software Engineer to build core systems for its AI Gateway, a secure control point that helps developers take AI agents from prototype to production. The Gateway provides model and tool routing, an OpenAI-compatible interface and MCP server, observability, policy enforcement, and capabilities such as Auto-RAG, tool injection, and session summarization.

The role involves designing and implementing the model router, MCP gateway, and control plane; building infrastructure for dynamic model selection, failover, cost-based routing, and policy enforcement; and owning capabilities including credential storage, session summarization, caching, and rate limiting. The engineer will develop APIs for OpenAI-compatible and Model Context Protocol integrations and build infrastructure for evaluation, telemetry, replay, and backtesting of agents and LLM workflows.

The successful candidate will contribute to architecture, technical strategy, roadmap planning, and cross-functional alignment, while mentoring engineers as the team grows. The work addresses provider abstraction, secure and fast identity handling, agent visibility and enforceable policies, and low latency and high reliability at scale. Docker describes itself as a globally distributed, remote-first team and offers work-from-anywhere arrangements, with offices in Seattle and Paris.

Requirements

• At least 6 years of backend engineering experience with production-grade systems.
• Deep knowledge of distributed and highly scalable systems, cloud-native infrastructure, and API design.
• Experience building secure, high-throughput services such as gateways, proxies, load balancers, or policy engines.
• Fluency in Go and/or Rust.
• Familiarity with AI/ML platforms or model-serving infrastructure.
• A product mindset and enthusiasm for building developer-facing tools.
• Ownership mentality with a bias for shipping, learning, and iterating.
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Nice To Haves

• Experience with both Go and Rust.
• Prior experience with OpenAI, Anthropic, or similar LLM APIs.
• Familiarity with RAG architectures, vector databases, or agent frameworks such as LangChain, AutoGPT, or CrewAI.
• Experience with policy engines such as OPA, observability frameworks such as OpenTelemetry, or API gateways.
• Understanding of OAuth 2.1, secret management, and cloud IAM systems.
• Experience with Kubernetes, Docker, and microservices architecture.

Responsibilities

• Design and implement core AI Gateway systems, including the model router, MCP gateway, and control plane.
• Build infrastructure for dynamic model selection and routing, automatic failover, cost-based routing, and policy enforcement.
• Own secure credential storage, session summarization, caching, and rate limiting capabilities.
• Develop APIs for developers using OpenAI-compatible interfaces and the Model Context Protocol.
• Build infrastructure for evaluation, telemetry, replay, and backtesting of agents and LLM workflows.
• Lead architectural decisions and mentor engineers as the team scales.
• Tune performance, including caching, session summarization, and rate limiting, to keep the Gateway fast under production load.
• Contribute to roadmap planning, technical strategy, and cross-functional alignment.

Benefits

• Generous paid time off, designated quarterly Whaleness Days, and an end-of-year Whaleness break.
• Home office support.
• Technology stipend equivalent to US$100 net per month.
• Annual stipend for conferences, courses, certifications, and continued learning.
• 16 weeks of paid parental leave after six months of employment.
• Equity for all full-time employees.
• Medical benefits, retirement benefits, and paid holidays, varying by country.
• Docker swag.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior