Lead AI Engineer

Wells Fargo & Company•Iselin, NJ

19h

About The Position

This role is for a Lead Specialty Software Engineer within the COO Technology team, which is central to powering enterprise operations at scale, including strategic execution, resiliency, regulatory enablement, customer experience, supply chain, and shared services. The team focuses on large-scale modernization and building resilient, data-driven platforms for operational excellence. This is a hands-on engineering position that requires expertise in distributed systems, streaming data, and enterprise-scale AI. The successful candidate will be responsible for architecting and building production-grade AI pipelines, integrating modern observability into LLM and VLM (Vision Language Model) systems, and ensuring reliability, performance, and cost transparency from the outset. Key tasks include designing scalable ingestion and streaming orchestration for high-volume clickstream and case-lifecycle data, deploying and operating containerized services using CI/CD and infrastructure-as-code, and implementing rigorous monitoring, tracing, and resiliency patterns for AI systems. The role also involves working with video and transcript data for downstream analysis and seamless integration with evaluation and analytics workstreams, ultimately delivering innovative, trusted, and explainable AI platforms for business operations.

Requirements

5+ years of Specialty Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education.
2+ years of experience with streaming platforms for data orchestration.
2+ years of experience with containerization and container orchestration platforms
2+ years of deploying and operating ML/AI models in production environments.
3+ years of experience programming with Python.

Nice To Haves

5+ years of experience in software engineering with focus on data pipelines and distributed systems.
Experience with LLMOps/MLOps frameworks for observability.
Experience with Vision Language Models (VLMs) or multi-modal AI systems.
Experience with enterprise container platforms.
Background in video processing or computer vision pipelines.
Experience with observability tools and platforms.
Experience building resilient systems with patterns like circuit breakers, bulkheads, and retry policies.
Experience with cloud computing platforms.
Knowledge of infrastructure-as-code tools.
Experience with CI/CD pipelines.
Understanding of data quality and validation frameworks.
Experience in financial services or enterprise environments.
Excellent communication skills across technical and non-technical audiences.

Responsibilities

Pipeline Development: Build scalable, robust ingestion pipelines for processing clickstream data, handling both participant-level (full-day employee activity) and case-level (end-to-end case lifecycle) data streams.
Streaming Orchestration: Design and implement streaming platform-based orchestration for pipeline coordination, ensuring reliable data flow and processing guarantees.
Container Platform Deployment: Deploy and manage containerized services on enterprise container platforms, implementing CI/CD pipelines and infrastructure-as-code practices.
AI Model Observability: Implement comprehensive observability for LLM and VLM pipelines, including: Performance monitoring and metrics collection. Distributed tracing for multi-model pipelines. Logging and alerting for model inference. Cost tracking and optimization.
Resiliency Engineering: Build fault-tolerant systems with retry mechanisms, circuit breakers, dead letter queues, and graceful degradation patterns.
Video/Transcript Processing: Work with VLMs to process clickstream video data and generate high-quality transcripts for downstream analysis.
Integration: Ensure seamless integration with downstream Analysis & Evaluation workstream.