Principal AI Engineer

Ontrac Solutions•San Francisco, CA

About The Position

We are seeking a Principal GenAI Architect / Forward Deployed Principal Engineer to lead the design, delivery, and productionization of enterprise-grade Generative AI solutions within a highly regulated banking environment. This role will partner directly with business, product, engineering, architecture, risk, cyber, legal, compliance, and model governance teams to translate high-impact financial services use cases into scalable, secure, and compliant AI systems. The ideal candidate has deep experience designing and delivering production-scale GenAI solutions across enterprise environments, with strong exposure to banking, financial services, risk management, customer operations, regulatory controls, and data protection requirements. This person will serve as a recognized GenAI expert across the organization, helping define reusable patterns, influence platform roadmaps, and raise the technical bar for responsible AI adoption across lines of business.

Requirements

7+ years of software engineering experience, with significant depth in AI/ML, data-intensive systems, distributed systems, or enterprise-scale platforms.
2+ years of hands-on Python programming experience.
2+ years of experience designing and delivering production-scale Generative AI systems in an enterprise environment.
2+ years of experience with LLMs, prompt engineering, and RAG architecture.
2+ years of experience with vector databases, semantic search, and retrieval systems.
2+ years of experience with API-driven, cloud-native architectures.
2+ years of experience with distributed systems, performance optimization, scalability, and reliability engineering.
Experience working within banking, financial services, fintech, or another highly regulated enterprise environment.

Nice To Haves

Strong understanding of banking technology environments and non-functional requirements, including security, scalability, resiliency, observability, latency, auditability, and cost management.
Demonstrated Principal-level impact, with influence across multiple teams, platforms, business units, or lines of business.
Experience operating in highly regulated environments, with financial services or banking experience strongly preferred.
Hands-on experience with agentic AI frameworks, AI workflow orchestration, and multi-model routing.
Prior experience in a Forward Deployed Engineer, embedded engineering, or business-facing technical delivery model.
Experience partnering with risk, cyber, legal, compliance, data governance, or model governance teams to deliver production AI solutions.
Familiarity with banking controls, customer data protection, model risk management, audit requirements, regulatory expectations, and responsible AI principles.
Ability to communicate complex technical concepts clearly to senior executives, technical stakeholders, risk partners, compliance teams, and non-technical business leaders.

Responsibilities

Own and define enterprise-grade GenAI architectures for banking and financial services use cases, including RAG pipelines, agentic workflows, prompt orchestration, and multi-model routing strategies.
Drive production readiness for GenAI solutions, ensuring scalability, resiliency, observability, latency optimization, security, and cost efficiency.
Lead complex architectural decisions across data ingestion, vector databases, model selection, guardrails, API scalability, performance optimization, and secure integration with banking systems.
Establish reference architectures, reusable patterns, and best practices to accelerate responsible GenAI adoption across lines of business.
Partner with central AI platform teams to align solution architecture with enterprise AI services, platform capabilities, banking technology standards, and long-term roadmap priorities.
Act as a Forward Deployed Principal Engineer, partnering directly with business, product, and engineering teams to deliver high-impact GenAI use cases from ideation through production.
Translate banking and financial services business problems into clear AI system designs, technical requirements, and non-functional requirements with measurable outcomes.
Support GenAI use cases across areas such as customer service, operations, knowledge management, risk, compliance, fraud, employee productivity, document intelligence, and internal workflow automation.
Troubleshoot and resolve complex issues across non-production and production environments.
Partner closely with application teams to ensure AI solutions integrate effectively into existing banking platforms, enterprise workflows, data environments, APIs, and control frameworks.
Influence platform roadmap by feeding real-world banking use-case requirements back into central AI services and enterprise AI platform teams.
Ensure all AI solutions align with banking risk, cyber, model governance, data protection, privacy, and regulatory expectations.
Partner with Cyber, Model Risk Management, Legal, Compliance, Risk, and Data Governance teams to design compliant AI patterns without slowing delivery.
Embed security, privacy, responsible AI, ethical AI, explainability, human oversight, and auditability into solution design by default.
Help define practical implementation patterns that balance innovation, speed, governance, regulatory expectations, and enterprise control requirements.
Ensure GenAI solutions are designed with appropriate guardrails for sensitive financial data, customer information, personally identifiable information, and regulated business processes.
Serve as a recognized GenAI expert across the enterprise, regularly consulted by engineering, architecture, product, risk, compliance, and leadership teams.
Mentor senior and staff-level engineers and help raise the technical bar across banking technology teams.
Contribute to internal communities of practice, architecture reviews, executive-level technical discussions, AI governance forums, and knowledge-sharing sessions.
Represent enterprise GenAI capabilities in internal innovation forums, banking technology discussions, and responsible AI showcases.