Principal Engineer, LLM

Upstart
•$238,400 - $330,200•Remote

About The Position

The Principal Engineer, LLM is a technical pathfinder and force multiplier for Upstart’s Applied AI team and partner product squads. The role focuses on shortening the path from ambiguous ideas to reliable, user-facing AI systems—not on owning company-wide developer-tooling infrastructure or training foundation models from scratch. The role has an explicit 80/20 charter. Approximately 80% is devoted to prototyping high-ambiguity product and architecture initiatives across the agent platform, cross-agent context orchestration, borrower-facing assistant experiences, and internal workflows. This includes partnering with Product and Design to validate non-deterministic user experiences, establish architectural patterns, and create evaluation contracts that engineering squads can use to scale successful work. Approximately 20% is devoted to serving as an AI-native engineering practitioner on the product side, modeling agentic coding and verification practices, adopting internal tools early, and helping product squads work more effectively. The engineer will treat user experience as part of AI systems architecture and navigate trade-offs involving latency, unit economics, reliability, and user trust. The role is remote, with the team operating across East and West Coast time zones. Employees are encouraged to participate in team onsites, which most teams hold once or twice per quarter for two to four consecutive days. The position’s anticipated U.S. remote base salary range is $238,400–$330,200, with actual pay depending in part on location, skills, experience, and education or training. Compensation also includes target bonuses and equity. Benefits for full-time employees include health and financial coverage, paid leave, family support, wellness and productivity allowances, and other resources.

Requirements

  • At least 8 years of software engineering experience.
  • Hands-on experience shipping non-deterministic, user-facing LLM or AI products to real users.
  • Practical experience designing production LLM systems using structured outputs, tool calling, context engineering, retrieval or memory patterns, and multi-step or multi-agent orchestration, including state management and context sharing.
  • Strong distributed-systems and backend fundamentals for non-deterministic pipelines, including asynchronous orchestration, streaming user experiences, caching, timeouts, fallback paths, rate-limit handling, and end-to-end observability or tracing.
  • Experience building golden datasets, automated evaluation harnesses using deterministic checks, heuristics, and LLM-as-a-judge, regression gates, and systematic failure-mode analyses.
  • Sound judgment in applying deterministic logic, prompting, context engineering, and guardrails against hallucinations and prompt or data risks, and in determining when routing, fine-tuning, or distillation is justified.
  • Ability to prototype across backend and frontend stacks, such as Python and TypeScript/React, and modern REST or streaming APIs.
  • Track record of aligning early with partner squads and handing off ownership cleanly, while partnering with Product, Design, and operations or business leaders to turn ambiguous goals into concrete latency budgets and resilient fallback experiences.
  • Bachelor’s degree in Computer Science, Engineering, Mathematics, or equivalent practical experience.

Nice To Haves

  • Experience in startup or low-boundary environments, owning ambiguous problems end to end under real constraints.
  • Experience designing reusable agent primitives, SDKs, or orchestration platforms that help engineering teams build, evaluate, and launch AI workflows safely and quickly.
  • Hands-on production experience fine-tuning or distilling smaller models to improve latency, unit economics, or task reliability.
  • Daily fluency with agentic coding workflows and automated verification harnesses, and experience helping peer engineers adopt higher-leverage development practices.
  • Experience in regulated or high-trust consumer domains such as fintech, lending, or healthcare, where accuracy, explainability, PII protection, and user trust are critical.

Responsibilities

  • Rapidly prototype ambiguous initiatives across multi-agent context sharing, user experiences for non-technical users, agent-platform primitives, and borrower-connected internal workflows.
  • Create durable handoff materials for production engineering, including golden datasets, automated evaluation harnesses, failure-mode maps, latency and cost benchmarks, and architectural blueprints.
  • Align early with receiving squad technical leads, surface system constraints, support go/no-go decisions with Engineering and Product leadership, and remain engaged through initial productionization, including reviewing early pull requests and debugging model edge cases.
  • Design non-deterministic user experiences and orchestration together, addressing latency spikes, context loss, hallucinations, and fallback paths to protect borrower and operator trust.
  • Choose pragmatic applied AI approaches, starting with the simplest reliable solution and making architectural decisions about model routing, fine-tuning, or distillation when warranted by latency, cost, or reliability.
  • Model disciplined agentic coding and verification practices, provide practical feedback to internal tooling teams, and help raise engineering judgment and applied AI practices across product squads.
  • Partner with Product, Design, engineering, and operations or business leaders to turn ambiguous goals into concrete requirements and resilient user experiences.

Benefits

  • Target bonuses and equity compensation, including annual equity grants that vest quarterly.
  • 401(k) or Group Retirement Savings Plan with a company match of $2 for every $1 contributed, up to $15,000 annually.
  • Employee Stock Purchase Plan with discounted stock purchase options for eligible U.S. employees.
  • Medical, dental, and vision coverage, plus wellness resources; Canadian employees receive supplemental health coverage.
  • Health Savings Account contributions for eligible U.S. plans.
  • Life insurance and disability coverage.
  • Paid time off, sick leave, and company holidays in line with local requirements.
  • Paid family and parental leave; duration varies by country.
  • Family-centered fertility, parenthood, and caregiving benefits.
  • Employee Assistance Program providing mental health support and life-centered resources.
  • Financial wellness resources, including financial planning tools and a financial concierge service for U.S. employees.
  • Annual wellness allowance for physical and emotional well-being and personal development.
  • Annual productivity allowance for relevant work tools and resources.
  • Team events, all-company updates, and employee resource groups.
  • Catered lunches and stocked micro-kitchens at offices in the Bay Area, Austin, Columbus, and New York City.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service