Senior AI Ops Engineer - Infrastructure

USAA•San Antonio, TX

9d•Onsite

About The Position

The Infrastructure AI Ops team owns AI-enabled operations across the application estate’s compute, network, storage, cloud, and on-premises platforms. The Senior AI Ops Engineer – Infrastructure will build high-throughput pipelines for infrastructure telemetry, develop models to baseline normal capacity and resource behavior and detect anomalies before they cause outages, and create automation to remediate infrastructure issues safely. This is a from-scratch build within a regulated financial institution, so every automated action must include a kill switch, a rollback path, and a rationale acceptable to risk examiners. The role is based in San Antonio, Texas, with an onsite requirement of five days per week. Relocation assistance is not available. The engineer will instrument and monitor models and automations in production, retrain models on a defined cadence, and participate in the on-call rotation for live actions at autonomy level 3 or above.

Requirements

Bachelor's degree, or four years of relevant education and/or experience.
At least five years of hands-on infrastructure, platform, or site-reliability engineering experience.
Strong programming and scripting ability, including Python and Go or a comparable systems language.
Practical experience building and productionizing machine-learning models for operational data, such as anomaly detection, forecasting, or capacity modeling, using TensorFlow, PyTorch, scikit-learn, or comparable frameworks.
Deep hands-on experience with infrastructure-as-code and orchestration tools such as Terraform, Ansible, Kubernetes operators, or equivalent, for production compute, network, and storage environments.
Experience writing and testing operational automation for infrastructure changes, including rollback or compensating actions and approval-gated or policy-controlled execution.
Ability to operate in a regulated environment and debug and reason about a live production infrastructure incident under time pressure.

Nice To Haves

Experience designing, deploying, and operationalizing model-driven capacity management or anomaly-detection capabilities for infrastructure operations and on-call engineering teams.
Experience developing guardrailed production-infrastructure automation, including automated scaling, node replacement, or storage-volume expansion, with tested kill switches and safety controls.
Experience designing and implementing operational AI evaluation frameworks, including test cases, regression suites, and drift detection to identify model or threshold degradation before production deployment.
Experience building or maintaining runtime service or infrastructure topology graphs and reconciling topology data with systems of record such as a CMDB.
Familiarity with enterprise AIOps and observability platforms such as Moogsoft, BigPanda, or Dynatrace.
Experience partnering directly with model governance or model risk functions, including developing model cards and supporting documentation for model validation, risk assessment, and governance review.

Responsibilities

Build and operate high-throughput ingestion and feature pipelines that bring compute, network, storage, container, and cloud-platform telemetry from the Observability & Telemetry team into the AIOps feature store.
Train, validate, and tune unsupervised and time-series models to baseline normal infrastructure behavior and detect capacity and resource anomalies before they become incidents.
Build a dynamic service and infrastructure topology graph using CMDB, discovery, and live signals, and design event-correlation logic to reduce infrastructure alerts reaching a human.
Write, test, and version remediation automation for infrastructure failure modes using Terraform, Ansible, or equivalent tools, with a tested rollback or compensating action for every change.
Operate within approval-gate, blast-radius, and kill-switch controls for infrastructure automation.
Instrument and monitor models and automations in production, and retrain models on a defined cadence.
Participate in the on-call rotation for live actions at autonomy level 3 or above.