Software Engineer, LLVM Compiler - MTIA

Meta•Menlo Park, CA

5d•$183,997 - $257,000

About The Position

Meta’s MTIA Software team builds the software stack for Meta’s in-house AI accelerator family, including the compiler and toolchain, runtime, kernel libraries, developer tooling, and deep PyTorch integration. The team co-designs this stack with silicon teams across hardware generations. The GenAI Frameworks organization makes frontier large language, multimodal, and diffusion models run efficiently on MTIA in production. The Software Engineer will own a key area of GenAI inference on MTIA, spanning serving frameworks, distributed inference, graph-mode execution and compilation, or core PyTorch integration. This domain-expert role sets technical direction, leads ambiguous multi-quarter programs across team boundaries, and owns quality across accuracy, stability, performance, and test coverage on custom silicon. The role includes taking framework and integration features from prototype through production, optimizing inference workloads across framework, runtime, compiler, and kernel layers, enabling and validating models against GPU baselines, and partnering with kernel, compiler, and silicon teams on hardware/software co-design.

Requirements

Bachelor’s degree in Computer Science, Computer Engineering, a relevant technical field, or equivalent practical experience.
8+ years of professional experience in systems software, ML infrastructure, performance engineering, or compiler/framework development.
5+ years of hands-on experience with generative AI inference or training frameworks, or production model-serving systems.
Proficiency in Python, C++, or Rust, including low-level systems code and performance-critical paths.
Experience with GenAI inference optimization, including prefill and decode optimization, KV-cache management and compression, batching and scheduling, and latency/throughput tradeoffs.
Hands-on experience with deep learning framework internals beyond model authoring, such as operator registration and dispatch, eager execution, autograd, or graph capture and compilation paths.
Experience with runtime-level optimization, including graph-mode execution, host-side latency reduction, and memory allocation and placement.
Demonstrated experience leading technical design and end-to-end delivery of frameworks or infrastructure projects, including test strategies and CI, across team boundaries.
Experience working across multiple system-stack layers, such as framework, compiler, runtime, and hardware, and root-causing issues across component boundaries.
Experience debugging numerical accuracy issues and maintaining accuracy and stability standards, not only optimizing performance.

Nice To Haves

Experience with LLM inference serving stacks, including continuous batching, paged or chunked attention, speculative decoding, prefix caching, disaggregated prefill and decode, or mixture-of-experts routing and load balancing.
Experience implementing responsible and ethical AI practices, such as risk assessment, bias mitigation, and quality and accuracy reviews.
Experience with long-context inference techniques, including context and sequence parallelism, ring or streaming attention, and hierarchical or offloaded KV cache.
Experience with AI accelerator platforms such as GPUs, TPUs, or custom ASICs, and bringing up a new hardware backend in a major machine learning framework.
Experience optimizing performance in production environments, using profiling and tracing to identify and resolve bottlenecks.
Demonstrated ability to integrate AI tools to optimize or redesign workflows and deliver measurable impact, such as efficiency gains or quality improvements.
Experience with distributed inference or training at scale, including tensor, pipeline, and expert parallelism, collective communication primitives, RDMA transports, multi-host serving, and scale-up and scale-out design.
Experience with low-precision numerics and quantization, including FP8, block-scaled formats, INT8/INT4, calibration, accuracy error analysis, and debugging tools for distributed systems.
Demonstrated ongoing AI skill development, such as prompt/context engineering or agent orchestration, and staying current with emerging AI technologies.
Familiarity with the PyTorch compilation stack, including TorchDynamo, FX IR, Torch Inductor, dynamic shapes, Triton, AOT I, and custom operator integration.

Responsibilities

Serve as technical owner and domain expert for a key GenAI inference framework area, such as serving runtime, distributed inference, graph-mode execution and compilation, or core PyTorch integration.
Lead ambiguous, multi-quarter technical programs end to end, including technical design, execution, test strategy and CI, rollout, and production hardening across teams and organizational boundaries.
Design, implement, and ship generative AI inference framework features and MTIA integration layers from prototype through production deployment.
Own end-to-end performance for production inference workloads by profiling across framework, runtime, compiler, and kernel boundaries and driving fixes to the appropriate layer.
Build and extend distributed inference capabilities, including hierarchical KV caching, cross-host transfer paths, disaggregated prefetch and decode, and parallelism strategies for long-context and mixture-of-experts serving.
Optimize runtime and execution paths, including graph capture and replay, host-side latency, memory allocation and placement, and throughput under real traffic conditions.
Enable frontier models on MTIA and validate accuracy against GPU baselines, closing correctness gaps and distinguishing real regressions from stale references.
Maintain quality for the area across accuracy, stability, latency, throughput, and test coverage, with regression detection to prevent performance gains from eroding.
Partner with kernel, compiler, and silicon teams on hardware/software co-design, providing framework-level evidence that can inform silicon designs.
Mentor engineers, improve design quality through reviews, and communicate decisions through design documents and cross-team reviews.