Principal Engineer AI

Pennymac • Westlake Village, CA

2d • Onsite

About The Position

As the Principal Applied AI Engineer, you will be the lead architect of the "cognitive" layer of the AI platform. While the Backend Principal builds the deterministic infrastructure, you will engineer the probabilistic systems that power our "Unified Context Library" and "Agentic Orchestration" layer. You will design the autonomous workflows that turn a Product Manager’s idea into a technical specification, and a User Story into deployable code. This role sits at the intersection of elite software engineering and applied artificial intelligence. You will not be training foundational models from scratch; rather, you will be mastering the art of Applied AI: orchestration, retrieval-augmented generation (RAG), and the engineering of agentic systems using the AWS Bedrock AgentCore framework. You will work within the AI Platform Services division, a dedicated R&D unit tasked with delivering measurable velocity improvements to the entire organization. Why Join Pennymac? ● Architect the Future of SDLC: You will build the "Agent Factory" that drives our "conveyor belt" of software delivery, moving us from a reactive to a proactive engineering culture. ● Greenfield Innovation: This is a rare chance to build an enterprise-grade AI platform from the ground up, leveraging the latest in AWS Bedrock AgentCore and Agentic AI frameworks. ● High-Impact & Visibility: Your work will directly impact the daily lives of hundreds of engineers and product owners, reducing "Idea-to-MR" cycle times and eliminating manual toil. ● Cutting-Edge Stack: Work with a modern, cloud-native stack (AWS, Node.js/TypeScript) specifically tailored for high-performance AI applications.

Requirements

• Elite Engineering Core: Bachelor’s Degree in Computer Science or equivalent, with 8+ years of professional software engineering experience. You are a software engineer first, who has mastered AI tools.
• TypeScript/Node.js Expert: Unlike most AI roles that focus on Python, our platform is built on Node.js and TypeScript. You must have deep expertise in building backend services and AI chains in this ecosystem.
• Applied AI & Agent Experience: Hands-on experience building applications powered by LLMs. You have shipped products using frameworks like LangChain, Strands, or AWS Bedrock.
• RAG Mastery: Proven track record of building production-grade RAG systems. You understand the nuances of embeddings, vector stores (Pinecone, Milvus), and semantic search.
• Cloud Native (AWS): Extensive experience with AWS serverless architecture (Lambda, API Gateway, DynamoDB). Familiarity with AWS Bedrock and AgentCore is a significant advantage.
• Systems Thinking: Ability to design complex, asynchronous systems where state is fluid and outcomes are probabilistic.
• Startup Mentality: High ownership, high energy, and the ability to thrive in a fast-paced "internal startup" environment.

Nice To Haves

• Experience with Evaluation Frameworks (e.g., LangSmith, Ragas) for automated testing of LLM outputs.
• Familiarity with the Model Context Protocol (MCP) for standardizing AI tool connections.
• Background in Developer Tools (building CLI tools, IDE plugins, or CI/CD automations).
• Understanding of Graph Databases (e.g., Neo4j) for knowledge graph implementation alongside vector search.

Responsibilities

• Architect Agentic Workflows
• Design and implement sophisticated multi-agent systems that can plan, execute, and self-correct complex tasks (e.g., automated code reviews, test plan generation, and epic decomposition).
• Develop robust orchestration flows using LangChain.ts and AWS Bedrock AgentCore, defining how agents hand off tasks to one another and when to loop in humans for review.
• Engineer "hallucination checkpoints" and validation logic to ensure AI outputs are accurate, secure, and deterministic where necessary.
• Implement the Model Context Protocol (MCP) to standardize how our agents interface with internal tools like Jira, GitLab, and AWS infrastructure.
• Build the Unified Context Library (RAG)
• Lead the strategy for our Retrieval-Augmented Generation (RAG) foundation. You will design the pipelines that ingest, chunk, and vectorize institutional knowledge from Confluence, Jira, and GitLab.
• Optimize Vector Database performance (e.g., Pinecone, Weaviate) and implement advanced retrieval strategies (hybrid search, re-ranking) to ensure agents possess the precise, domain-specific context needed for mortgage-tech tasks.
• Implement "memory" systems (Short-term and Long-term) that allow agents to retain context across long-running sessions and provide personalized assistance to users.
• AI System Engineering & Observability
• Design and maintain the Observability & Fine-Tuning Framework, ensuring we capture every token, prompt, and user feedback signal (thumbs up/down) to systematically improve agent performance over time.
• Define and enforce Prompt Engineering best practices, creating a reusable library of system prompts that govern agent persona, tone, and output formatting.
• Build automated Evaluation Pipelines (using tools like LangSmith or custom harnesses) to benchmark agent performance against "Golden Datasets" and prevent regression.
• Technical Leadership
• Serve as the subject matter expert on Generative AI for the Platform Services division, staying ahead of the curve on LLM capabilities, cost optimization, and model selection (e.g., routing tasks between Claude 3.5 Sonnet, GPT-4o, and smaller, faster models).
• Mentor fellow engineers on the paradigm shift from deterministic coding to probabilistic AI engineering.
• Drive the adoption of AI best practices across the wider organization.

Benefits

• Comprehensive Medical, Dental, and Vision
• Paid Time Off Programs including vacation, holidays, illness, and parental leave
• Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)
• Retirement benefits, life insurance, 401k match, and tuition reimbursement
• Philanthropy Programs including matching gifts, volunteer grants, charitable grants and corporate sponsorships

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Number of Employees: 1,001-5,000 employees