Manager - AI Inference Engineer

EY•Hartford, DC

1d•$125,500 - $261,600•Hybrid

About The Position

We are seeking an experienced AI Inference Engineer to help design, deploy, and operate private AI inference infrastructure at rack scale. This role is focused on the engineering of a production-grade inference platform: architecture, serving stack design, benchmarking, integration, reliability, and operational readiness. The ideal candidate has deep hands-on experience building AI inference systems and the platform layers that support them, including multi-GPU inference, Kubernetes-based deployment, autoscaling, observability, routing, access control, and production support.

Requirements

Bachelor’s Degree in a relevant field
Strong experience designing and operating AI inference systems in production.
Deep familiarity with multi-GPU serving and high-performance deployment patterns.
Hands-on experience with Kubernetes and cloud-native platform technologies.
Strong understanding of scaling, routing, autoscaling, observability, and reliability for production AI workloads.
Experience integrating AI platforms into enterprise security and access environments.
Ability to turn business and workload needs into concrete infrastructure and software design decisions.
Clear technical communication and strong architecture documentation skills.

Nice To Haves

Experience with private, sovereign, or enterprise AI infrastructure.
Experience benchmarking large-model inference on GPU clusters.
Familiarity with service mesh, policy enforcement, secrets management, and RBAC.
Experience working across infrastructure, security, and application teams in a regulated environment.
Experience supporting pilot-to-production AI platform rollouts.

Responsibilities

Design and deploy secure private inference architectures for enterprise AI inference.
Translate workload requirements into technical architecture decisions across compute, caching, interconnect, storage & networking.
Define reference architectures that support high-throughput, low-latency, enterprise inference at scale.
Design the model-serving stack, including inference runtime, orchestration, model registry, artifact management, API layer, routing, observability, deployment and fallback.
Evaluate and recommend inference technologies and platform components for production use.
Configure and harden the target environment for secure, reliable inference workloads.
Build and run benchmark suites for enterprise workloads.
Measure latency, throughput, concurrency, utilization, reliability, and cost efficiency.
Tune serving configurations and system parameters to improve production performance.
Produce evaluation results and recommendations based on objective testing.
Support integration of inference platforms with client IT systems.
Work with infrastructure and security stakeholders to validate technical requirements and remediate issues.
Onboard models into the target inference platform.
Configure serving endpoints, routing behavior, monitoring, and operational controls.
Support deployment readiness, runbooks, and operational handoff for production use.