Senior Software Engineer II - Ads Quality

Instacart

13d•$192,000 - $242,500•Remote

About The Position

As a Senior Software Engineer II on the Ads Quality team, specializing in ML infrastructure, you will work with software and machine learning engineers to build systems that optimize ad performance across the serving funnel, from retrieval to pricing. You will own technical direction for data, feature processing, and model training and serving solutions, supporting new modeling approaches including LLMs and bespoke foundation models. You will address bottlenecks in training and serving to improve stability, latency, and cost. The role focuses on designing, developing, and deploying machine learning infrastructure for a complex marketplace, in partnership with the company-wide ML Foundation team. You will build real-time feature processing infrastructure and the foundation for a real-time sequential model, including online sequence feature stores, streaming update paths, efficient long-context serving, and training pipelines. You will collaborate with product managers, data scientists, and ML engineers to co-design solutions around business needs, product requirements, model architecture, and infrastructure constraints. You will also mentor engineers, set technical direction, and conduct code reviews. Instacart Ads aims to improve advertising outcomes through relevance, fairness, and performance. The Ads Quality team uses large-scale data and machine learning for response prediction, advertiser optimization, and ad auctions, and is exploring LLM capabilities. This position is remote under Instacart’s Flex First work approach, which allows employees to work from home, an office, or another preferred location, with regular in-person events.

Requirements

Graduate degree (master’s or PhD) in computer science, artificial intelligence, machine learning, operations research, or equivalent self-study and experience.
At least 3 years of industry experience building ML infrastructure or using machine learning to solve real-world problems with large datasets.
Deep expertise building, scaling, and managing production ML infrastructure for training and/or serving, with a strong understanding of modern ML systems and MLOps.
Strong software engineering programming skills, such as Python or Go.
Expertise in data manipulation using SQL, Spark, and Pandas, and familiarity with ML frameworks such as Torch, TensorFlow, and XGBoost.
Strong analytical and problem-solving skills.
Excellent verbal and written communication skills and ability to collaborate with diverse stakeholders across all levels.
Willingness to work in unfamiliar domains, languages, and problem areas.

Nice To Haves

At least 5 years of industry experience tech leading a team building ML infrastructure or applying machine learning to real-world problems with large datasets.
Demonstrated experience solving complex model training and serving efficiency challenges, including optimizing training pipelines, reducing training times, and improving serving architectures.
Hands-on experience with real-time or online feature processing and serving, including streaming systems such as Kafka or Flink, online feature stores, and low-latency retrieval on the request path.
Experience with sequential modeling, transformer architecture, generative retrieval, or large-scale online recommendation systems.
Experience serving large sequential or foundation models in low-latency, high-QPS production settings.
Experience in digital advertising platforms.
Familiarity with LLM integrations, prompt engineering, and productivity tooling.

Responsibilities

Design, develop, and deploy machine learning infrastructure to address marketplace challenges and advance the training and serving platform.
Work with the company-wide ML Foundation team to develop infrastructure solutions for Ads-specific needs.
Profile and optimize training and serving bottlenecks to improve throughput, cost, and latency, including accelerator utilization and serving tail latency.
Identify system and operational weaknesses and improve availability.
Build low-latency real-time feature processing pipelines that compute, join, and serve fresh features for ad requests.
Lead infrastructure development for a real-time sequential foundation model, including online sequence feature stores, streaming update paths, efficient long-context serving, and training pipelines.
Collaborate with product managers, data scientists, and machine learning engineers to understand business needs and co-design systems.
Engage with stakeholders to ensure solutions are integrated and aligned with business goals.
Mentor engineers, set technical direction, and conduct code reviews to support code quality and maintainability.
Promote continuous improvement and technical excellence.