Senior Site Reliability Engineer AI Security Federal

BizFirst • McLean, VA

7d • Onsite

About The Position

BizFirst is assisting its client with recruiting a Senior Site Reliability Engineer focused on AI security in a federal civilian agency environment. The role operates and maintains a self-managed, distributed AI runtime security platform on AWS EKS. Because the platform handles agency AI requests, its availability and latency are important to mission systems.

The engineer will work alongside the agency's DevOps and security teams to operate, scale, and upgrade the platform, and partner with the client's engineering team to improve reliability over time. The client is a customer-centric, specialized integrator with an engineering-first culture focused on IT modernization, performance engineering, and cybersecurity. Its teams work as trusted advisors in a collaborative, fast-paced environment that supports innovation, continuous improvement, professional growth, and work-life balance.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• Five years of experience in site reliability, DevOps, or cloud infrastructure engineering.
• Experience operating production Kubernetes clusters, including monitoring, scaling, and troubleshooting containerized workloads.
• Experience with core AWS services, including IAM, VPC, EC2, load balancing, KMS, Secrets Manager, and CloudWatch.
• Experience managing infrastructure as code with Terraform.
• Experience scripting or programming in Python, Go, or Bash.
• Ability to obtain and maintain a DHS Public Trust determination.

Nice To Haves

• Experience running Amazon EKS, including managed node groups, Karpenter or Cluster Autoscaler, IRSA or EKS Pod Identity, and the AWS Load Balancer Controller.
• Experience operating latency-sensitive, high-throughput services, including load testing and performance tuning.
• Experience with Prometheus, Grafana, OpenTelemetry, and forwarding logs to a SIEM such as Splunk.
• Experience with Helm and GitOps or CI/CD for Kubernetes, including Argo CD, Flux, GitHub Actions, or GitLab CI.
• Experience in federal environments, including ATO, NIST SP 800-53, AWS GovCloud, FIPS endpoints, and STIG-hardened container images.
• Experience with GPU node pools or serving machine learning models on Kubernetes.
• Familiarity with AI and LLM applications and traffic patterns, including streaming responses and large payloads.
• Certifications such as CKA, CKS, AWS Solutions Architect or SysOps Administrator, or HashiCorp Terraform Associate.

Responsibilities

• Operate, monitor, and troubleshoot a distributed, high-throughput platform on Amazon EKS, addressing pod and node issues, networking, storage, and performance bottlenecks.
• Plan and execute platform, Kubernetes, and add-on upgrades with minimal downtime while following agency change management.
• Manage infrastructure as code with Terraform and Helm to keep environments consistent from development through production.
• Design and tune autoscaling, capacity, and load testing to meet latency and throughput targets as demand grows.
• Build and maintain metrics, logs, traces, and alerting; define service level objectives (SLOs); and lead incident response and post-incident reviews.
• Implement customer security requirements, including secrets management, least-privilege IAM, network controls, image hardening, patching, and evidence for authorization to operate (ATO) and continuous monitoring.

Benefits

• Medical coverage.
• Dental coverage.
• Vision coverage.
• Employer matching contributions to a 401(k) retirement plan.
• Competitive compensation and bonus opportunities.
• Support for continuing education and professional growth, including tuition reimbursement.
• Paid leave and holidays.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior