Machine Learning Engineer 4 - Manager IC

Capital One • McLean, NY

2d

About The Position

Capital One is seeking a Machine Learning Engineer to help build and scale production AI and machine learning systems. As part of the Enterprise ML Platform team, the role contributes to a high-throughput, multi-tenant training platform that supports workflows ranging from classical machine learning to generative AI fine-tuning and distributed computing with Ray, Spark, and Dask. The team owns the Kubeflow Pipelines v2 application layer on Amazon EKS and develops internal Python and Go SDKs, custom Kubernetes operators, and developer tools for complex, critical, regulated workloads. The role also integrates agentic AI into engineering lifecycles and contributes to upstream open-source Kubeflow while supporting platform stability and security.

The position combines machine learning engineering with operations, modeling, and data engineering. It involves designing and delivering models and components, building cloud-based ML infrastructure and data pipelines, developing and testing application code, and deploying, maintaining, and monitoring production solutions. The engineer works with Product, Data Science, and cross-functional Agile teams, and applies responsible and explainable AI, risk governance, security, and software delivery best practices.

Requirements

• Bachelor's degree or higher in Computer Science, Machine Learning, or a related quantitative field such as Statistics, Economics, Operations Research, Analytics, Mathematics, or Engineering.
• At least 4 years of programming experience with Python, Java, Golang, or C++.
• At least 4 years of machine learning experience using industry-standard frameworks such as PyTorch or TensorFlow and libraries such as Pandas, NumPy, and Scikit-learn.
• At least 4 years of experience using and operating large-scale distributed systems such as Spark or Ray to prepare AI or machine learning data.
• At least 2 years of experience deploying and operating machine learning solutions in production, operating cloud production services on AWS, GCP, or Azure, and using Kubernetes to manage large-scale containerized ML software systems.

Nice To Haves

• Master's or doctoral degree in Computer Science, Electrical Engineering, Mathematics, or a related field.
• 3 or more years designing, training, optimizing, and evaluating modern ML models across architectures such as Transformers, CNNs, and RNNs, and paradigms such as supervised, unsupervised, and reinforcement learning, including loss design, hyperparameter tuning, and diagnostic debugging.
• 3 or more years architecting and scaling distributed, end-to-end ML systems, from production-ready training and evaluation data pipelines to algorithm and infrastructure optimization.
• 3 or more years managing ML workloads with Docker, Kubernetes, and pipeline orchestration tools; Kubeflow is a strong plus.
• 3 or more years applying software best practices such as CI/CD, version control, automated testing, and code reviews, and deploying resilient systems with progressive rollout strategies, monitoring, alerting, and incident response planning.
• At least 1 year of experience as a technical lead developing ML solutions using industry best practices, patterns, and automation.
• Authored or co-authored a paper on an ML technique, model, or proof of concept.

Responsibilities

• Design, build, and deliver machine learning models and components that address business problems in collaboration with Product and Data Science teams.
• Use knowledge of modeling techniques and issues—including model, data, and feature selection; training; hyperparameter tuning; dimensionality; bias and variance; and validation—to inform ML infrastructure decisions.
• Solve complex problems by writing and testing application code, developing and validating ML models, and automating tests and deployment.
• Collaborate on cross-functional Agile teams to create and enhance software for big data and machine learning applications.
• Retrain, maintain, and monitor models in production.
• Leverage or build cloud-based architectures, technologies, and platforms to deliver optimized ML models at scale.
• Construct optimized data pipelines to feed ML models.
• Apply continuous integration and continuous deployment practices, including test automation and monitoring, to support successful deployment of models and application code.
• Manage code to reduce vulnerabilities, govern models from a risk perspective, and follow Responsible and Explainable AI practices.
• Build and evolve the Enterprise ML Platform, including its Kubeflow Pipelines v2 application layer on Amazon EKS, internal SDKs, Kubernetes operators, and developer tooling.
• Integrate agentic AI into engineering lifecycles and contribute to upstream open-source Kubeflow.

Benefits

• Comprehensive, competitive, and inclusive health, financial, and other benefits supporting total well-being; eligibility varies by employment status and management level.
• Eligible for performance-based incentive compensation, which may include cash bonuses and/or long-term incentives.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior