Staff Data Engineer - Agentic AI

American Express•Palo Alto, CA

About The Position

At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. From delivering differentiated products to providing world-class customer service, we operate with a strong risk mindset, ensuring we continue to uphold our brand promise of trust, security, and service. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career. Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express. Joining Amex Tech means discovering and shaping your contribution to something big. Here, you can work alongside talented tech teams and build a unique career with the Powerful Backing of American Express. With a range of opportunities to work with the latest technologies, and a commitment to back the broader engineering community through open source, our mission is to power your success. Because Amex Tech is powered by our technology, our culture, and our colleagues. The Technology organization enables and accelerates the company’s growth strategies, delivering global capabilities and services in support of Amex’s customers and colleagues, while maintaining 24/7 servicing and availability to ensure an uninterrupted, high-quality customer experience. Technology provides the foundation for everything we do in the company while driving differentiation through building and leveraging innovative technology and data insights. At American Express, AI is reshaping the future of commerce and redefining the experiences our commercial customers and card members expect. Within Amex Technology, we are building platforms, products, and governance that enable agentic AI systems to operate responsibly and at scale across the enterprise. Our focus is on agentic AI development: designing intelligent, adaptive systems that can plan, reason, and act across complex workflows with appropriate levels of autonomy. These systems power autonomous workflows, decision support, and customer-facing experiences—while meeting the high standards for security, explainability, reliability, and compliance required in financial services. We partner closely with product, design, and business teams to deliver agentic capabilities that reduce operational friction, improve decision-making, and transform how customers interact, transact, and grow. The Role As a Staff Data Engineer – Agentic AI, you will operate as a senior individual contributor within Amex Technology, helping define how agentic AI systems are designed, built, and operated across the company. You will work at the intersection of agentic system architecture and hands-on execution: designing core frameworks, writing production code, reviewing critical designs, and guiding teams through complex technical decisions. This role does not include people management; your impact comes from technical leadership, sound judgment, and the ability to turn ambiguity into reliable, scalable agentic systems. You will work closely with Product and UX partners to shape a shared vision for agentic AI experiences. While product teams own prioritization, Staff engineers are expected to actively influence what gets built and how agentic systems come together across teams. This is an individual contributor, hands-on, deeply technical role with broad architectural ownership and high organizational impact; however, as the team grows, you will take on increasing responsibility for technical leadership and people development within the agentic AI space. Mentoring, developing and managing direct reports may be in scope for some of our opportunities.

Requirements

10+ years of experience building large-scale distributed systems + strong experience with LLM systems, agentic workflows or advanced ML infrastructure
Proven ownership of complex, cross-cutting agentic systems spanning multiple teams or products.
Strong engineering fundamentals across backend systems, APIs, data pipelines, and cloud infrastructure.
Deep experience across the agentic AI stack, including planning, tool use, memory, and evaluation.
Fluency with AI-assisted and agentic development workflows.
Comfort operating in ambiguous problem spaces and translating them into shipped, reliable autonomous systems.
Ability to influence technical direction and align teams without formal authority.
Experience in workflow engines, async processing, queues, and streaming systems.

Nice To Haves

Experience building agentic systems in fintech or other regulated industries.
Experience as a founding engineer or early technical leader in AI-driven products.
Demonstrated success delivering technically complex autonomous systems that customers actively rely on.
Meaningful contributions to open-source AI or agentic frameworks.
Familiarity with fine-tuning, model optimization and inference pipelines is a plus

Responsibilities

Drive technical direction for agentic AI initiatives, influencing architecture patterns, autonomy boundaries, and system design.
Design, build, and operate production-grade agentic AI systems used across multiple products.
Own and evolve shared agentic AI capabilities, including:
Agent frameworks and orchestration layers
Planning, tool use, and memory strategies
Retrieval and grounding (RAG) pipelines
LLM infrastructure, inference, and model gateways
Evaluation, observability, and safety tooling for autonomous systems
Lead technical design reviews and help teams navigate tradeoffs involving autonomy, safety, reliability, scalability, and cost.
Partner across teams to deliver complex, cross-cutting agentic AI initiatives from concept to production.
Evaluate emerging models, techniques, and agentic patterns and translate them into practical, enterprise-ready improvements.
Mentor senior engineers and raise the technical bar for agentic AI development through example and influence.

Benefits

Competitive base salaries
Bonus incentives
6%25 Company Match on retirement savings plan
Free financial coaching and financial well-being support
Comprehensive medical, dental, vision, life insurance, and disability benefits
Flexible working model with hybrid, onsite or virtual arrangements depending on role and business need
20+ weeks paid parental leave for all parents, regardless of gender, offered for pregnancy, adoption or surrogacy
Free access to global on-site wellness centers staffed with nurses and doctors (depending on location)
Free and confidential counseling support through our Healthy Minds program
Career development and training opportunities