Senior AI Ops Engineer - Application

USAA • San Antonio, TX

1d • $114,080 - $218,030 • Onsite

About The Position

The Senior AI Ops Engineer will build and operate application-focused AI operations capabilities at USAA, connecting machine learning to the services, APIs, and transaction flows that support customer accounts. This is a from-scratch build in a regulated financial institution, and the work will touch production systems under audit scrutiny.

The engineer will develop ingestion pipelines that turn application logs, traces, and business-transaction signals into features; train and tune models to identify degrading APIs and anomalous transaction failures; and build tested, deterministic automation to act on model findings. This includes guided remediation, safe auto-restarts, and rollbacks.

The position is based in San Antonio, Texas, with an onsite requirement of five days per week. Relocation assistance is not available. The role participates in an on-call rotation for live actions at autonomy level 3 or above.

Requirements

• Bachelor's degree, or four years of relevant education and/or experience.
• At least five years of hands-on software or data engineering experience.
• Strong programming ability in Python and a second production language, with experience delivering tested, code-reviewed production code.
• Practical experience building and productionizing machine-learning models for anomaly detection, classification, or forecasting using frameworks such as TensorFlow, PyTorch, or scikit-learn, including the lifecycle from training through retraining.
• Working fluency with LLM-based techniques, including prompt design, retrieval-augmented generation, and structured outputs, and understanding of their failure modes.
• Experience writing and testing operational automation, such as workflow orchestration or runbook-as-code, including rollback design and approval-gated or policy-controlled execution.
• Ability to operate in a regulated environment and debug and reason about a live production incident under time pressure.

Nice To Haves

• Proven experience deploying model-driven root-cause analysis or incident-triage solutions used in production application operations.
• Experience building production automation with tested guardrails, safety controls, and kill-switch capabilities.
• Experience developing AI evaluation and monitoring frameworks, including regression testing, model and prompt evaluation, and drift detection.
• Experience with event correlation, alert grouping, and high-volume application telemetry, including exposure to enterprise AIOps platforms such as Moogsoft, BigPanda, or Dynatrace.
• Experience working with model governance or model risk teams and supporting model validation with appropriate documentation, including model cards.
• US military experience gained through military service or as a military spouse or domestic partner.

Responsibilities

• Build and operate high-throughput ingestion and feature pipelines that consume application logs, traces, APM data, and business-transaction events collected by the Observability & Telemetry team and deliver them to the AIOps feature store.
• Train, validate, and tune anomaly-detection and forecasting models using application and transaction-level signals.
• Build LLM-based log parsing, summarization, and root-cause-hypothesis components with explicit handling for hallucination risk.
• Design and build event-correlation and probable-cause-ranking logic to reduce the volume of raw application alerts reaching human responders.
• Write, test, and version remediation automation for application-layer failure modes, including tested rollback for every action and operation within approval-gate, blast-radius, and kill-switch frameworks.
• Instrument and monitor models and automations in production and retrain models on a defined cadence.
• Participate in the on-call rotation for live actions at autonomy level 3 or above.

Benefits

• Comprehensive medical, dental, and vision plans.
• 401(k) and pension.
• Life insurance.
• Parental benefits and adoption assistance.
• Paid time off program with paid holidays.
• 16 paid volunteer hours.
• Wellness programs.
• Career path planning and continuing education.
• Employees may be eligible for pay incentives based on overall corporate and individual performance and at the discretion of the USAA Board of Directors.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior