Principal Machine Learning Engineer

CVS Health•New York, NY

23h•$144,200 - $288,400

About The Position

Join CVS Health’s Personalization & Decisioning Platform team to help build a unified, large-scale machine learning platform that drives individualized offer, content, and channel decisions for millions of customers. This is a hands-on, backend-heavy engineering role focused primarily on designing and building production systems rather than only developing models. As the team’s most senior engineer, you will set technical direction for designing, building, testing, deploying, and operating machine learning systems at scale. You will architect backend services, data contracts, and distributed pipelines that support causal inference, reinforcement learning, and large-scale optimization models in production. You will also raise engineering standards through design and code reviews, mentorship, and technical leadership. You will work closely with data engineering, MLOps, product, and platform teams to make models maintainable, observable, testable, and safe to change. The role includes leading complex initiatives, coaching mid-level and senior ML engineers, and helping shape the team roadmap. Experience in retail, ad-tech, or large-scale marketing and offer personalization is a strong plus. The representative technology stack includes Python, SQL, scikit-learn, XGBoost, pandas, NumPy, SciPy, optimization solvers, Dask, Kubeflow Pipelines, Kubernetes, Flask, SQLAlchemy, Snowflake, cloud object storage, Docker, CI/CD, and infrastructure-as-code. Deep expertise in every listed tool is not required; strong engineering fundamentals and the ability to ramp up are emphasized.

Requirements

At least 7 years of progressively complex software, machine learning, or data science/engineering experience, including at least 5 years building and operating production backend systems or data-intensive platforms at scale.
At least 5 years building, deploying, and owning production machine learning systems, with a strong focus on software development practices such as testing, CI/CD, code review, and modular design.
Expert-level Python and SQL skills.
Deep understanding of software design patterns, API design, and distributed systems fundamentals.
Proven experience architecting scalable, data-intensive backend systems—including microservices, REST APIs, and batch or streaming pipelines—in a major cloud environment such as Azure, AWS, or GCP.
Hands-on experience with Docker and Kubernetes, and with ML/data pipeline orchestration tools such as Kubeflow, Airflow, or similar.
Demonstrated technical leadership, including setting technical direction, conducting design and code reviews, and mentoring engineers.
Strong track record of anticipating risks and preventing production issues, and communicating technical tradeoffs clearly to engineering teams and business stakeholders.
Bachelor’s degree or equivalent work experience in Computer Science, Mathematics, Statistics, Engineering, Business Analytics, Economics, Physics, or a related discipline.

Nice To Haves

Experience with large-scale mathematical or combinatorial optimization, including MIP solvers such as Gurobi, PuLP, or SCIP, in a production decisioning system.
Experience with causal inference, including uplift modeling, CATE estimation, or meta-learners, and/or reinforcement learning applied to real-world decisioning or recommendation problems.
Experience designing versioned feature stores with point-in-time correctness and training-serving parity.
Experience with distributed computing frameworks such as Dask, Spark, or Ray for large-scale data processing and model training.
Experience in retail, ad-tech, or marketing personalization.
Proficiency across mathematical analysis methods, machine learning, and statistical/predictive modeling, with deep specialization in at least one area such as causal inference, reinforcement learning, or optimization.
Master’s degree; preferred but not required with sufficient hands-on production experience.

Responsibilities

Architect and build scalable, production-grade backend services, including APIs, microservices, and batch and streaming pipelines, to serve machine learning decisions under strict latency, throughput, and reliability requirements.
Design data contracts and interfaces among data engineering, model training, and production serving layers, emphasizing training-serving parity and point-in-time correctness.
Own the design of versioned, queryable feature stores and point-in-time lookup APIs that prevent feature leakage and support training and real-time inference through a single code path.
Design and optimize distributed computing workflows, including containerized batch jobs, parallelized training, and optimization solves, to meet operational SLAs.
Make and document build-versus-buy and architecture tradeoffs involving storage formats, orchestration frameworks, solvers, and serving patterns.
Set and enforce engineering standards for modular design, dependency management, code reviews, automated testing, CI/CD, and documentation.
Champion practices that enable safe changes to ML systems, including contract tests, reproducible environments, infrastructure-as-code, and rollback-safe deployments.
Drive observability and monitoring for software and model health, including service-level metrics, data and feature drift, model calibration, and automated alerting.
Lead design and code reviews, balance pragmatism with maintainability, and prevent or resolve technical debt.
Lead complex cross-repository initiatives by breaking ambiguous problems into phased, de-risked implementation plans.
Mentor mid-level and senior machine learning engineers on system design, modeling tradeoffs, and production readiness.
Anticipate scaling limits, failure modes, operational risks, and potential roadblocks in systems under design.
Partner with engineering managers and product leadership to shape the roadmap, estimate complex work, and raise technical standards across the organization.
Communicate technical concepts, tradeoffs, and risks to engineering peers and non-technical business stakeholders.
Build and productionize large-scale decisioning systems, including causal inference and incrementality models, reinforcement-learning-based policies, and mixed-integer optimization for constrained allocation.
Work with data science teams to turn research prototypes into reliable, scalable production systems.
Build and maintain experimentation infrastructure, including holdouts, A/B tests, and counterfactual validation, and monitor incrementality, drift, and model and business performance.