Forward Deployed Engineer, Inference

Cohere•San Francisco, CA

11d•$240,000 - $415,000

About The Position

Cohere is hiring a Staff Forward Deployed Engineer to bridge its frontier AI models and complex enterprise environments. The role owns architecture for ambiguous, customer-specific inference challenges, helping strategic partners deploy Cohere models reliably, securely, and with high performance within their infrastructure and data perimeters. The engineer serves as a principal technical liaison between Cohere’s core engineering teams and customers, leading technical adoption end to end and optimizing latency, throughput, and resilience. The role involves working directly with customer engineering teams to integrate models into production environments, designing advanced serving and failover architectures, optimizing private deployments, benchmarking customer workloads, and validating models on emerging hardware. It also includes translating customer needs into product improvements and helping customers establish reliable production deployments, service-level objectives, and observability practices.

Requirements

At least 8 years of engineering experience running production infrastructure, designing large-scale distributed systems, or deploying enterprise software in complex environments.
Deep production Kubernetes expertise, including multi-cluster orchestration and hybrid or multi-cloud ecosystems such as GCP, Azure, AWS, and OCI.
Strong understanding of LLM architecture and inference dynamics, including attention mechanisms and the performance characteristics of prefill and decode phases.
Hands-on experience with modern inference engines and optimization stacks such as vLLM, TensorRT-LLM, lmd, or Dynamo.
Familiarity with distributed serving techniques, including prefill/decode disaggregation, distributed KV caching, and multi-GPU workloads.
Exceptional customer-facing technical leadership, including scoping ambiguous problems and guiding executive technical stakeholders.
High agency, grit, and adaptability, including ability to write production code quickly while navigating customer deployment blockers.
Strong programming skills in Python, Go, or C++, with the ability to work across backend data pipelines and infrastructure tuning.

Responsibilities

Embed with customer engineering teams to design integrations between Cohere AI models and complex production environments.
Design model-serving topologies, including multi-tenant architectures, intelligent model routing, and cross-datacenter failover for multi-cluster environments.
Optimize private and “AI-in-a-box” deployments, balancing Kubernetes scale and reliability constraints with model performance.
Benchmark and profile customer LLM workloads, diagnosing performance issues using metrics, dashboards, and distributed logs.
Support customer validation of Cohere models on GPUs, TPUs, AMD accelerators, and custom ASICs.
Translate customer deployment challenges into platform upgrades and contribute feedback to core product features.
Help customers establish production deployment patterns, SLOs, and observability frameworks.

Benefits

Weekly lunch stipend of $75/£75 or local equivalent.
Health and dental benefits, including a separate mental health budget.
RRSP matching, 401(k), and pension scheme.
100% parental leave top-up for up to six months for either parent.
Annual enrichment benefits for arts and culture, fitness and wellness, quality time, and workspace improvements.
Education and learning stipend for conferences, courses, and coaching.
Six weeks of paid vacation (30 working days).
Travel budget for remote employees to visit other offices, plus an annual company offsite.
Daily lunch program, snacks, and community and social events for office-based employees.
Co-working benefit for employees not near an office.
$500 home office stipend.