Principal AI Platform and Model Operations Engineer - TS/SCI with Polygraph

GD Information Technology • Chantilly, VA

3d • Onsite

About The Position

GDIT is seeking a hands-on AI Platform Operations Engineer to own and mature the end-to-end platform and lifecycle mechanisms that move AI models, prompts, retrieval assets, dependencies, and services from approved development into reliable, classified production environments. The role works across AI engineering, cloud and platform engineering, security, data, and Service Operations to ensure advanced AI capabilities can be deployed and operated safely at scale in highly regulated environments.

The engineer will design and manage a controlled model and software supply chain, implement local inference services, and orchestrate repeatable release trains for AI workloads. The role establishes standards for observability, capacity management, and safe rollback in environments where imports, deployments, and configuration changes are governed events. The position is onsite in Chantilly, Virginia and requires an active TS/SCI clearance with polygraph and U.S. citizenship.

Requirements

• Bachelor’s degree in computer science, software engineering, machine learning, information technology, or a related field, or approved equivalent experience.
• Active TS/SCI clearance with polygraph.
• U.S. citizenship.
• At least 8 years of platform, DevSecOps, SRE, ML engineering, or cloud engineering experience, including at least 3 years supporting production AI/ML platforms.
• Hands-on experience deploying and operating local model-serving platforms such as vLLM, TGI, Triton, TensorRT-LLM, llama.cpp, Ollama, ONNX Runtime, or comparable technologies.
• Strong Linux, container, Kubernetes/OpenShift, networking, storage, GPU, identity, secrets, and infrastructure automation skills.
• Experience with model and artifact registries, experiment tracking, release management, CI/CD, internal package and container repositories, and disconnected dependency management.
• Ability to size inference capacity using model precision, VRAM, context length, concurrency, throughput, latency, and availability requirements.
• Strong Python and automation proficiency, plus experience with Terraform, Ansible, Helm, GitLab CI/Jenkins, or comparable tooling.
• Working knowledge of RMF/STIG, software supply-chain security, SBOMs, vulnerability management, logging and audit, and classified data handling.
• Demonstrated technical leadership, incident judgment, cross-team coordination, mentoring, and clear documentation.

Nice To Haves

• Experience with PyTorch, Hugging Face, MLflow, Kubeflow, Ray, NVIDIA GPU tooling, CUDA, or enterprise AI platform products.
• Experience operating A100, H100, L40S, or comparable accelerators and performing quantization or optimization.
• Experience with Azure Government, AWS GovCloud, C2S-equivalent, or on-premises platform delivery.
• Role-relevant CKA/CKS, current Red Hat platform engineering credentials, or Microsoft Certified: DevOps Engineer Expert.
• Relevant Terraform, NVIDIA, or ML-platform credentials.
• Experience integrating model-quality and evaluation signals with deployment gates and production monitoring.

Responsibilities

• Design and operate AI/ML platform services for local model serving, orchestration, model and artifact registries, experiment and evaluation tracking, feature or retrieval assets, and deployment automation.
• Build controlled import and release workflows for model weights, containers, Python packages, libraries, prompts, datasets, and vulnerability updates in disconnected environments.
• Implement provenance, checksums and signing, SBOMs, malware and vulnerability scans, license review, allowlists, approvals, staged validation, and rollback for AI artifacts.
• Deploy and tune inference services using approved serving stacks; manage batching, concurrency, memory, quantization, autoscaling where available, queueing, resilience, and graceful degradation.
• Establish versioning and release relationships among models, tokenizers, prompts, retrieval indexes, embedding models, configurations, evaluation sets, applications, and infrastructure.
• Develop GPU, CPU, storage, and network capacity models and advise on hardware options, consolidation, scheduling, and cost/performance trade-offs.
• Implement health, latency, throughput, saturation, failure, quality, drift, and usage telemetry integrated with enterprise observability and ServiceNow.
• Define backup and recovery, continuity, patching, vulnerability treatment, certificate and secret rotation, and incident procedures for AI platform services.
• Partner with AI Assurance and Security to enforce use-case gates, evaluation thresholds, audit, data restrictions, and release evidence.
• Lead technical reviews, troubleshoot production AI platform incidents, mentor engineers, and document reusable platform patterns.

Benefits

• Comprehensive benefits and wellness packages.
• 401(k) with company match.
• Competitive pay and paid time off.
• Medical plan options, some with Health Savings Accounts.
• Dental plan options and vision plan.
• Full flex work weeks where possible.
• Vacation, sick and personal time, holidays, paid parental, military, bereavement, and jury duty leave.
• Short- and long-term disability benefits.
• Life, accidental death and dismemberment, personal accident, critical illness, and business travel and accident insurance.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal