Senior AI Ops Engineer - Infrastructure

USAA•San Antonio, TX

1d•$114,080 - $218,030•Onsite

About The Position

USAA is hiring a Senior AI Ops Engineer for its Infrastructure AI Ops team, which owns AI-enabled operations across compute, network, storage, cloud, and on-premises platforms. The engineer will build telemetry ingestion and feature pipelines, develop models to baseline infrastructure behavior and detect anomalies, and create automation to remediate infrastructure issues. This is a from-scratch build within a regulated financial institution, where automated actions must have a kill switch, a tested rollback path, and a rationale suitable for risk review. The role is based in San Antonio, Texas, and requires working onsite five days per week. The work environment is described as flexible, but the stated onsite requirement is five days weekly. Relocation assistance is not available. The engineer will instrument and monitor models and automations in production, retrain models on a defined cadence, and participate in the on-call rotation for live actions at autonomy level 3 or above.

Requirements

Bachelor’s degree, or four years of relevant education and/or experience.
At least five years of hands-on infrastructure, platform, or site-reliability engineering experience.
Strong programming and scripting ability, including Python and Go or a comparable systems language.
Practical experience building and productionizing machine-learning models for operational data, such as anomaly detection, forecasting, or capacity modeling, using TensorFlow, PyTorch, scikit-learn, or similar frameworks.
Deep hands-on experience with infrastructure-as-code and orchestration tools such as Terraform, Ansible, Kubernetes operators, or equivalent, in production compute, network, and storage environments.
Experience writing and testing operational automation for infrastructure changes, including rollback, compensating actions, and approval-gated or policy-controlled execution.
Comfort operating in a regulated environment and demonstrated ability to debug and reason about a live production infrastructure incident under time pressure.

Nice To Haves

Experience designing, deploying, and operationalizing model-driven capacity management or anomaly-detection capabilities for infrastructure operations and on-call engineering teams.
Experience developing guardrailed automation for production infrastructure, including automated scaling, node replacement, or storage volume expansion, with tested kill-switch and safety-control mechanisms.
Experience designing and implementing operational AI evaluation frameworks, including test cases, regression suites, and drift detection to identify model or threshold degradation before production deployment.
Experience building or maintaining runtime service or infrastructure topology graphs and reconciling topology data with systems of record such as a CMDB.
Familiarity with consuming outputs from enterprise AIOps and observability platforms such as Moogsoft, BigPanda, or Dynatrace.
Experience partnering with model governance or model risk functions, including developing model cards and supporting model validation, risk assessment, and governance review.

Responsibilities

Build and operate high-throughput ingestion and feature pipelines that bring compute, network, storage, container, and cloud platform telemetry into the AIOps feature store.
Train, validate, and tune unsupervised and time-series models to establish infrastructure behavior baselines and detect capacity and resource anomalies.
Build a dynamic service and infrastructure topology graph from CMDB, discovery, and live signals, and design event-correlation logic to reduce infrastructure alerts reaching human responders.
Write, test, and version infrastructure remediation automation using Terraform, Ansible, or equivalent tools, with tested rollback or compensating actions and within approval-gate, blast-radius, and kill-switch frameworks.
Instrument and monitor models and automations in production, retrain models on a defined cadence, and participate in the on-call rotation for live actions at autonomy level 3 or above.