AI/ML Operations & Assurance Engineer

Peraton•Chantilly, VA

4d•Onsite

About The Position

Peraton is seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. The role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and safely introducing AI-related changes into production. The engineer collaborates with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services. This position is fully onsite in Chantilly, Virginia. The engineer monitors production AI services and workflows, develops operational evaluations, manages AI-related configuration and changes, and supports security, audit, and compliance activities. The role also identifies operational improvements, measures workflow impacts, reports service and usage trends, and recommends enhancements to monitoring, testing, configuration control, and operational procedures.

Requirements

Active TS/SCI clearance with CI Polygraph.
BS/BA and 5 years of experience; an additional 4 years of experience may be considered in lieu of a degree.
Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.
Experience operating and troubleshooting production software applications or cloud services.
Experience with monitoring, logging, metrics, incident response, and configuration or change management.
Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis.
Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.
Experience with AWS or comparable cloud platforms.
Strong analytical, troubleshooting, documentation, and communication skills.

Nice To Haves

Experience with AWS Bedrock or comparable managed foundation-model services.
Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing.
Experience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies.
Familiarity with ServiceNow, IT service management, and incident-management processes.
Experience with GitLab, CI/CD pipelines, and controlled configuration promotion.
Familiarity with Kubernetes and containerized application environments.
Experience monitoring cloud utilization, model consumption, and service costs.
Knowledge of risk management frameworks, continuous monitoring, or security evidence collection.
Experience supporting classified, restricted, or highly regulated environments.

Responsibilities

Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.
Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.
Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.
Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.
Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.
Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.
Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.
Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production.
Analyze user feedback and operational results to identify workflows that need refinement, additional controls, or restrictions on automated execution.
Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions.
Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations.
Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders.
Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality.
Ensure AI-related changes follow established review, testing, approval, and deployment procedures.
Monitor configuration drift and verify production services remain aligned with approved configurations.
Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services.
Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns.
Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews.
Verify production workflows use approved models, endpoints, tools, and data sources in accordance with established policies.
Support implementation and validation of operational safeguards for AI-enabled functions.
Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.
Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency.
Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand.
Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues.
Recommend improvements to monitoring, testing, configuration control, and operational procedures.