Software Engineer II- AI/ML, AWS Neuron

Amazon • Seattle, WA

1d

About The Position

The Annapurna Labs team at Amazon Web Services (AWS) builds AWS Neuron, the
software development kit used to accelerate deep learning and GenAI workloads on
Amazon’s custom machine learning accelerators, Inferentia and Trainium.

The AWS Neuron SDK, developed by the Annapurna Labs team at AWS, is the backbone
for accelerating deep learning and GenAI workloads on Amazon's Trainium ML
accelerators. This comprehensive toolkit includes an ML compiler, runtime, and
application framework that seamlessly integrates with popular ML frameworks like
PyTorch and JAX enabling unparalleled ML inference and training performance.

The Training Enablement and Foundation team is at the forefront of running a
wide range of models and supporting novel architecture alongside maximizing
their performance for AWS's custom ML accelerators. Working across the stack
from PyTorch till the hardware-software boundary, our engineers build systematic
infrastructure, innovate new methods and create high-performance kernels for ML
functions, ensuring every compute unit is fine tuned for optimal performance for
our customers' demanding workloads. We combine deep hardware knowledge with ML
expertise to push the boundaries of what's possible in AI acceleration.

As part of the broader Neuron organization, our team works across multiple
technology layers - from frameworks and kernels and collaborate with compiler to
runtime and collectives. We not only optimize current performance but also
contribute to future architecture designs, working closely with customers to
enable their models and ensure optimal performance. This role offers a unique
opportunity to work at the intersection of machine learning, high-performance
computing, and distributed architectures, where you'll help shape the future of
AI acceleration technology

You will architect and implement business critical features, and mentor a
brilliant team of experienced engineers. We operate in spaces that are very
large, yet our teams remain small and agile. There is no blueprint. We're
inventing. We're experimenting. It is a very unique learning culture. The team
collaborates with open source ecosystems to provide seamless integration and
bring peak performance at scale for customers and developers.

This role is responsible for development, enablement and performance tuning of a
wide variety of LLM model families, including massive scale large language
models like the Llama family, DeepSeek and beyond.

Requirements

• 3+ years of non-internship professional software development experience, or Bachelor's degree or above in engineering or equivalent
• 3+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience
• Experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution
• Knowledge of system performance, memory management, and parallel computing principles
• Experience in debugging, profiling, and implementing software engineering best practices in large-scale systems, or experience debugging, profiling, and implementing best software engineering practices in large-scale systems

Nice To Haves

• Experience with PyTorch, JIT compilation, and AOT tracing, or experience programming with at least one software programming language
• Experience with CUDA kernels or ML/low-level kernels, or experience in computer architecture
• Experience with distributed training at scale (FSDP, Tensor Parallelism, Expert Parallelism)

Responsibilities

• Design, develop, and optimize machine learning models and frameworks for deployment on custom ML hardware accelerators.
• Participate in all stages of the ML system development lifecycle including distributed computing based architecture design, implementation, performance profiling, hardware-specific optimizations, testing and production deployment.
• Build infrastructure to systematically analyze and onboard multiple models with diverse architecture.
• Analyze and optimize system-level performance across multiple generations of Neuron hardware
• Conduct detailed performance analysis using profiling tools to identify and resolve bottlenecks
• Conduct comprehensive testing, including unit and end-to-end model testing with continuous deployment and releases through pipelines.
• Work directly with customers to enable and optimize their ML models on AWS accelerators
• Collaborate across teams to develop innovative optimization techniques

Benefits

• health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• paid time off
• parental leave

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level