Principal AI Software Engineer

Walker & Dunlop

31d•$250,000 - $300,000•Remote

About The Position

Walker & Dunlop is hiring a Principal AI Software Engineer for WDTech’s AI Development Team. The role will help shape the company’s Loan Origination and Servicing platform and the AI systems used to build, deploy, and operate software across the business. The engineer will design scalable applications, intelligent workflows, and agentic systems that provide real-time insights and improve operational efficiency for teams and clients. The position also contributes to Atlas, WDTech’s AI collaboration platform for reusable skills, agents, and AI capabilities. It calls for deep technical expertise, sound judgment, the ability to solve complex problems independently, and a commitment to delivering reliable solutions and raising engineering standards through collaboration and mentorship.

Requirements

12+ years of software development experience, including experience as a senior individual contributor delivering production systems.
Bachelor’s or master’s degree in Computer Science or a related field, or equivalent experience.
Extensive experience delivering production AI/ML systems, including LLM-powered applications and agentic workflows.
Deep experience with modern agent frameworks and SDKs such as Claude Agent SDK, LangGraph, and Pydantic AI, as well as MCP and A2A.
Experience with prompt and context engineering, including RAG, embeddings, hybrid search, and reranking.
Experience with model evaluation, observability, and guardrails for production AI systems.
Experience using agentic development tools, including skills, subagents, and harness configurations.
Deep proficiency in Python and modern backend frameworks such as FastAPI, with a focus on scalable and maintainable architecture.
Strong expertise with SQL databases, including schema design, query optimization, and performance tuning.
Experience testing non-deterministic systems, including LLM call fixtures, evaluation suites, and strategies for controlling flaky behavior.
Strong judgment in determining which capabilities belong in a shared platform versus individual applications.
Ability to operate effectively in ambiguity, establish emerging patterns, and independently move from business problem definition through solution delivery.
Experience with Event Sourcing and Domain-Driven Design principles.
Excellent verbal and written communication skills.
Ability to show ownership, take on challenges, acknowledge growth opportunities, and demonstrate patience when learning new processes.
Courtesy, respect, and thoughtfulness when working with colleagues and other stakeholders.

Nice To Haves

Experience in Real Estate, Financial Services, or Capital Markets.

Responsibilities

Own complex AI and platform initiatives end-to-end, delivering business-facing capabilities while strengthening the underlying context and agent platform.
Design, build, and maintain agentic workflows for business-critical processes such as loan intake, document review, and servicing.
Establish platform patterns and capabilities by defining approaches, validating them through production use cases, and scaling them for broader adoption.
Lead model selection and routing, evaluating models based on cost, latency, and quality, and evolving solutions as models and capabilities become available.
Design interfaces and capabilities for other teams, including MCP tool servers, agent handoffs, subagents, long-running state, and human-in-the-loop checkpoints.
Establish evaluation and reliability practices, including evaluation sets, LLM-as-judge approaches where appropriate, regression testing, and ongoing performance measurement.
Work with minimal oversight, identify risks proactively, make sound technical trade-offs, escalate when needed, and raise engineering standards through collaboration and mentorship.
Perform other duties as assigned.

Benefits

Opportunity to join a Fortune Magazine Great Place to Work winner.
Comprehensive benefit options, including medical coverage with up to 83% subsidized medical payroll deductions.
Competitive dental and vision benefits.
401(k) with company match.
Pre-tax transit and commuting benefits.
Health and wellness program with cash rewards and resources promoting health, engagement, and balance.
Paid maternity and parental leave, as well as other family paid leave programs.
Company-paid life insurance and short- and long-term disability insurance.
Health Savings Account and Healthcare and Dependent Care Flexible Spending Accounts.
Career development opportunities.
Volunteer hours and donation matching.