Senior Lead Data & AI Engineer

Vanguard•Malvern, PA

4d•Hybrid

About The Position

At Vanguard, we don't just have a mission—we're on a mission. To work for the long-term financial wellbeing of our clients. To lead through product and services that transform our clients' lives. To learn and develop our skills as individuals and as a team. From Malvern to Melbourne, our mission drives us forward and inspires us to be our best. Vanguard has implemented a hybrid working model for the majority of our crew members, designed to capture the benefits of enhanced flexibility while enabling in-person learning, collaboration, and connection. We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience. Vanguard, one of the world's leading investment management companies, serves individual investors, institutions, employer-sponsored retirement plans, and financial professionals. We have a diverse and talented crew with a culture that promotes teamwork, along with an unwavering focus on serving our clients' best interests.

Requirements

9–12+ years in data engineering, specializing in AI deployment within cloud ecosystems (Databricks, AWS).
Hands-on experience deploying AI agents using frameworks like RAG, graph RAG, and orchestrating agents (AgentOps, Agent-bricks, etc.).
Proficient in AWS AI/ML services (SageMaker, Bedrock) and orchestration tools (MWAA, Step Functions).
Strong knowledge of lakehouse architecture, Unity Catalog, and data modeling best practices.
Deep experience in data orchestration, monitoring, and scalable AI-driven workflows.

Responsibilities

Architect and build reusable, metadata-driven data and AI engineering frameworks that standardize ingestion, transformation, feature engineering, and AI workflow deployment.
Leverage Databricks, lakehouse architecture, declarative pipelines, and cloud-native services to enable scalable, governed, and reusable data products across the organization.
Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks to automate data processing, AI pipelines, and agent data refresh cycles.
Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments across multiple business workflows.
Integrate Lakeflow for real-time data stream processing, ensuring AI agents are updated and responsive to live data.
Ensure seamless orchestration between AI models and data pipelines, using event-driven architectures for real-time inference and deployment.
Implement and orchestrate AI agents using frameworks such as Agentic systems, AgentOps tooling, and solutions like Agents on Databricks (Agent-bricks).
Hands-on experience deploying AI agents using RAG, Graph RAG, MCP-enabled integrations, and agent orchestration frameworks such as AgentOps, AgentBricks, LangGraph, or cloud-native orchestration services.
Manage AI agent lifecycles, monitoring, and scaling using tools like SageMaker, Bedrock, or AI orchestration frameworks on AWS.
Ensure robust data governance, metadata management, and AI observability through Unity Catalog, AWS Glue, or custom metadata layers.
Design for scalability and modularity, ensuring AI agents are reusable across multiple business processes.