ML Ops Engineer

ICF•Reston, AL

4d•$98,614 - $167,644•Remote

About The Position

ICF's Digital Modernization Division is seeking an ML Ops Engineer to join a team accelerating the modernization of a large federal agency's enterprise data and analytics ecosystem. This cloud-based platform provides data storage, analytics, governance, and AI/ML capabilities. The role involves developing and maintaining machine learning pipelines, model registries, deployment frameworks, monitoring solutions, and automated workflows to support enterprise AI initiatives. The engineer will collaborate with data scientists, data engineers, cloud architects, platform engineers, and Trustworthy AI stakeholders to operationalize machine learning capabilities in cloud environments.

Requirements

U.S. Citizenship is required due to federal contract requirements.
Candidate must reside in the U.S., be authorized to work in the U.S., and all work must be performed in the U.S.
Candidate must have lived in the U.S. for three (3) full years out of the last five (5) years.
Bachelor's degree in Computer Science, Software Engineering, Machine Learning, Data Science, Engineering, or a related field or a high school diploma with four (4) additional years of relevant experience in lieu of a bachelor's degree.
6+ years of experience designing and supporting production machine learning solutions in cloud environments.
Master's degree may substitute for two (2) years of relevant experience.

Nice To Haves

Experience with machine learning operations (MLOps), automated deployment pipelines, and model lifecycle management.
Proficiency in Python and modern software development practices.
Experience with source control systems and CI/CD frameworks.
Hands-on experience with Azure Machine Learning, Databricks, MLflow, or comparable ML platforms.
Experience implementing model monitoring, automated retraining, and performance tracking solutions.
Knowledge of cloud-native architecture, containerization, and scalable model serving patterns.
Experience supporting Responsible AI, AI governance, or Trustworthy AI initiatives.
Familiarity with infrastructure-as-code and automated environment provisioning.
Strong understanding of cloud cost optimization practices for AI/ML workloads.

Responsibilities

Design, build, and maintain production-grade MLOps pipelines for model training, evaluation, validation, registration, deployment, monitoring, retraining, rollback, and retirement.
Develop reusable MLOps and LLMOps patterns using Azure Databricks, Databricks ML, MLflow, Unity Catalog, Azure Machine Learning, Python, and approved Azure AI services.
Implement experiment tracking, model versioning, model registries, lineage, quality metrics, model cards, and approval workflows using MLflow and Unity Catalog.
Build automated CI/CD pipelines using GitHub Actions, Azure DevOps, Databricks Asset Bundles, and Terraform.
Deploy and maintain scalable model-serving infrastructure for batch, streaming, real-time, and API-based inference workloads.
Establish automated testing and evaluation gates for model performance, data quality, fairness, safety, explainability, robustness, privacy, and release readiness.
Implement model and feature monitoring, drift and bias detection, logging, tracing, alerting, and operational dashboards.
Develop automated retraining, champion/challenger promotion, rollback, and model-retirement workflows.
Integrate ML workflows with source control, infrastructure provisioning, secrets management, identity and access controls, environment management, and enterprise release processes.
Implement Trustworthy AI controls, including model documentation, impact assessments, audit logging, content safeguards, human oversight, and evaluation evidence aligned with Government-approved standards and the NIST AI Risk Management Framework.
Partner with data scientists, AI engineers, data engineers, platform engineers, DevSecOps, Security, and Governance personnel to make models repeatable, testable, deployable, and operationally supportable.
Monitor compute and serving utilization, tune cluster and endpoint configurations, apply tagging conventions, and implement cost-optimization strategies.
Troubleshoot failed pipelines, deployment issues, model-serving incidents, performance degradation, and monitoring alerts.
Create and maintain deployment templates, reusable repositories, operational runbooks, model documentation, release evidence, and technical standards.