Senior Software Engineer II - Ads Quality

Instacart

6d•$192,000 - $242,500•Remote

About The Position

As a Senior Software Engineer II in the Ads Quality team specializing in ML infrastructure, you'll work with a team of software and machine learning engineers to build state-of-the-art systems that optimize Ads performance throughout the ads serving funnel, from retrieval to pricing. You'll own the technical direction of our data, feature processing and model training/serving solutions, enabling next-generation modelling opportunities—from utilizing out of the box LLMs to building bespoke Foundation Models that powers all the Ads use cases. You will eliminate bottlenecks across the training and serving system to optimize stability, latency and cost. Building on Instacart's unique data and market position, these systems create value for advertising partners, improve consumer experience, and directly contribute to Instacart's overall revenue / business success. You'll have the unique opportunity to collaborate with a cross-functional team, turning diverse insights into powerful solutions that elevate advertising effectiveness and operational stability. Your expertise will steer the development of scalable training and serving infrastructure, driving performance and efficiency gains across our advertising platform. Together, we'll build an ads ecosystem that not only serves millions but also excels in delivering value to users and advertisers alike. Instacart Ads is transforming the online grocery shopping experience by seamlessly integrating advanced advertising solutions into our platform. The Ads Quality team is dedicated to delivering outstanding advertising outcomes by driving innovation and precision in response prediction, advertiser optimization, and ads auction processes. Our mission is to ensure relevance, fairness, and performance for all participants in the ad ecosystem. By leveraging terabytes of data and sophisticated machine learning algorithms, we build advanced predictive models that enhance user experience, drive advertiser success, and sustain a thriving ad ecosystem. We're also at the forefront of integrating Large Language Models (LLMs), unlocking powerful new capabilities that are redefining how brands connect with consumers in the digital grocery space. Our next major bet is a real-time sequential foundation model for ads—and building the training and serving infrastructure to make it possible is central to this role.

Requirements

Have a graduate degree (masters or PhD) in computer science, artificial intelligence, machine learning, operations research or equivalent self study and experience.
Have 3+ years of industry experience building ML infrastructure or using machine learning to solve real-world problems with large datasets.
Deep expertise building, scaling and managing ML infrastructure for training and/or serving in production, with a strong understanding of modern machine learning systems and MLOps.
Strong programming skills in software engineering (e.g., Python, Go), along with expertise in data manipulation using tools (SQL, Spark, and Pandas), and ML frameworks (Torch, Tensorflow, XGBoost).
Strong analytical and problem-solving skills.
Excellent communication skills (both verbal and written) and can collaborate with a diverse group of stakeholders across all levels.
Eager to jump into domains, languages, and problem areas that might be new and unfamiliar.

Nice To Haves

Have 5+ years of industry experience tech leading a team to build ML infrastructure or using machine learning to solve real-world problems with large datasets.
Demonstrated experience solving complex model training and serving efficiency challenges, including optimizing training pipelines, reducing model training times, and improving serving architectures to enhance operational efficiency and scalability.
Hands-on experience with real-time / online feature processing and serving—streaming systems (e.g., Kafka, Flink), online feature stores, and low-latency retrieval on the request path.
Experience with sequential modeling, transformer architecture, and generative retrieval or building large scale online recommendation systems.
Experience serving large sequential or foundation models in low-latency, high-QPS production settings.
Experience in digital advertising platforms.
Familiarity with LLM integrations, prompt engineering, and productivity tooling.

Responsibilities

Design, develop, and deploy machine learning infrastructure to tackle practical challenges in our complex marketplace while advancing the training and serving platform.
Work with the company wide ML Foundation team to develop solutions to meet Ads unique needs.
Optimize training and serving funnel bottlenecks—profiling real systems and driving meaningful wins in throughput, cost, and latency, from GPU/accelerator utilization to tail latency at serving time.
Identify instabilities and shortcomings of our systems and operating procedures to deliver best in class availability.
Build real-time feature processing infrastructure—low-latency pipelines that compute, join, and serve fresh features to models on the critical path of every ad request.
Lead the infrastructure foundation for our real-time sequential foundation model, including online sequence feature stores, streaming update paths, efficient long-context serving, and training pipelines that keep pace with rapidly evolving data.
Collaborate closely with product managers, data scientists, and MLEs to deeply understand business needs and co-design systems where product, model architecture and infrastructure constraints are solved together.
Actively engage with diverse stakeholders to ensure that solutions are well-integrated and aligned with business goals.
Mentor engineers, set technical direction, and conduct thorough code reviews to ensure high code quality and maintainability. Lead by example in fostering a culture of continuous improvement and technical excellence.