Machine Learning Engineer- Inference Optimization | Experienced Hire
About The Position
We are looking for a Machine Learning Engineer focused on low-latency inference optimization to build, tune, and productionize high-performance model-serving systems. The role sits at the intersection of machine learning, systems engineering, and GPU performance, with an emphasis on inference workloads where latency, throughput, reliability, and hardware efficiency are important. The engineer will work closely with quantitative researchers and engineers to understand model structure, identify inference bottlenecks, and turn research ideas into efficient production systems. The work focuses on transformer-style architectures and structured inference workloads, while potentially involving other model types. The role evaluates and tunes inference frameworks and serving or compilation systems, while considering GPU execution, memory layout, batching strategies, precision tradeoffs, and end-to-end latency.
Requirements
- Experience deploying, optimizing, or operating machine-learning inference workloads in production or production-like environments.
- Programming experience in Python, Java, C#, or similar, and in at least one systems language such as C, C++, Rust, or Go.
- Solid understanding of modern ML frameworks such as PyTorch, including model execution, export, tracing, compilation, and performance profiling.
- Ability to reason about latency, throughput, batching, memory use, GPU utilization, and reliability under real workloads.
- Strong practical judgment about tradeoffs among model quality, latency, throughput, implementation complexity, and maintainability.
Nice To Haves
- Experience optimizing inference for latency-sensitive or high-throughput applications.
- Experience with model optimization techniques such as quantization, pruning, distillation, operator fusion, graph lowering, custom operators, or model compilation.
- Exposure to CUDA, Triton language, ROCm, PTX, CuTe, CUTLASS, FlashInfer, or similar low-level GPU programming tools.
- Experience running inference workloads on Kubernetes or GPU clusters, including scheduling, autoscaling, observability, and resource management.
- Background in mathematics, physics, computer science, engineering, statistics, quantitative finance, or another technical field.
- Demonstrated ability to improve real-world inference performance beyond a baseline framework implementation.
Responsibilities
- Design, build, and optimize low-latency inference systems for production machine-learning workloads.
- Profile inference pipelines across model execution, runtime configuration, batching, memory movement, serialization, networking, and I/O.
- Evaluate, integrate, and tune inference runtime systems.
- Improve latency, throughput, and GPU utilization for production inference workloads.
- Build and support benchmarking and profiling tools to compare model variants, hardware targets, runtime configurations, and deployment strategies.
- Debug performance issues involving GPU memory, compute saturation, kernel behavior, CPU/GPU coordination, data movement, and serving-layer overhead.
- Help shape model and system design choices so research models can be deployed efficiently under real latency constraints.
- Collaborate with lower-level systems or GPU specialists on custom operators, kernel-level optimization, or hardware-specific performance work when necessary.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
