Staff Software Engineer, AI Gateway

Docker

1d • $170,350 - $275,550 • Remote

About The Position

Docker is hiring a Staff Software Engineer to help build Docker AI Gateway, a secure control point for taking AI agents from prototype to production. The Gateway provides model and tool routing, an OpenAI-compatible interface and MCP server, observability, policy enforcement, and capabilities such as Auto-RAG, tool injection, and session summarization.

The engineer will set technical direction for core Gateway systems, including the model router, MCP gateway, and control plane, and ensure that architectural choices can scale. The role involves hands-on technical strategy and implementation, solving ambiguous infrastructure challenges, and partnering with engineering leadership, product, design, and other teams. The engineer will also serve as a technical expert and mentor, improving the team’s engineering standards and developer experience.

The work includes defining abstractions across model and tool providers; building secure identity and credential systems; enabling real-time and historical analytics; and ensuring policy enforcement and logging cover the full interaction from prompt through tool use to response. Docker describes the team as small, remote-first, and focused on building mission-critical infrastructure for agent-based applications.

Requirements

• At least 8 years of backend engineering experience and a record of owning production-grade systems at significant scale.
• Deep expertise in distributed systems, cloud-native infrastructure, and API design, recognized by peers and cross-functional partners as a go-to expert.
• Experience building secure, high-throughput services such as gateways, proxies, load balancers, or policy engines.
• Fluency in Go and/or Rust.
• A record of identifying systemic gaps in architecture, process, or tooling and driving fixes across team boundaries.
• A strong product mindset and experience shaping technical direction.
• Comfort operating autonomously with broad latitude to determine how to solve problems and when to involve others.
• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Nice To Haves

• Prior experience with OpenAI, Anthropic, or similar LLM APIs.
• Familiarity with RAG architectures, vector databases, or agent frameworks such as LangChain, AutoGPT, or CrewAI.
• Experience with policy engines such as OPA, observability frameworks such as OpenTelemetry, or API gateways.
• Understanding of OAuth 2.1, secret management, and cloud IAM systems.
• Experience with Kubernetes, Docker, and microservices architecture.
• A record of setting technical standards adopted by other teams or engineers.

Responsibilities

• Set technical direction for the AI Gateway’s model router, MCP gateway, and control plane, ensuring the architecture can scale.
• Identify gaps in infrastructure, process, or tooling and drive solutions from proposal through production.
• Help define, deliver, and implement the Gateway’s technical strategy in partnership with engineering leadership.
• Solve complex system challenges, including dynamic model selection, automatic failover, cost-based routing, and end-to-end policy enforcement.
• Serve as a technical expert on secure credential storage, session summarization, caching, and rate limiting.
• Build infrastructure for evaluating, replaying, and backtesting agent and LLM behavior.
• Mentor senior and mid-level engineers and raise technical standards through implementation, design reviews, and example.
• Partner with product, design, and leadership to shape the control-plane UI and product direction.
• Define a unified abstraction layer across model and tool providers such as OpenAI, Anthropic, Google, and AWS Bedrock.
• Architect secure, scalable identity and credential vaulting for tool and model access.
• Build infrastructure for real-time and historical analytics of AI agent behavior.
• Ensure policy enforcement and logging operate end-to-end from prompt to tool to response.
• Set a high standard for developer experience through intuitive APIs and first-class observability.

Benefits

• Generous paid time off, quarterly Whaleness Days, and an end-of-year Whaleness break.
• Home office support.
• Technology stipend equivalent to US$100 net per month.
• Annual learning and development stipend for conferences, courses, certifications, and continued learning.
• 16 weeks of paid parental leave after six months of employment.
• Equity for all full-time employees.
• Medical benefits, retirement benefits, and paid holidays, which vary by country.
• Docker swag.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior