Principal AI Engineer

Gallatin•Washington, DC
•$240,000 - $310,000

About The Position

Gallatin AI builds AI-driven logistics planning and decision support for military operations. The Principal AI Engineer will serve as a technical anchor for the growing AI team, setting engineering standards, owning architecture and scalability, and partnering closely with platform and backend teams so the team can design, build, review, and ship high-quality work. The role calls for a broadly skilled engineer with strong judgment who has designed systems other engineers build on, improves the capabilities of colleagues, and cares about how the team works as well as what it delivers. Deep AI experience is advantageous; deep engineering judgment is required. The Principal AI Engineer will guide architecture and engineering practices, contribute production code to difficult problems, and help the AI team work independently.

Requirements

  • 10 or more years of software engineering experience, including a record of designing and scaling complex production systems through multiple orders of magnitude of growth.
  • Principal-level depth in backend engineering and distributed systems, including service design, event-driven and message-based architectures, APIs, data modeling, concurrency, caching, and failure handling.
  • Demonstrated trust as a code and architecture gatekeeper for a team or platform, and a record of spreading that capability across other engineers.
  • Strong computer science fundamentals applied with sound judgment.
  • Deep fluency in Python and strong TypeScript or comparable full-stack experience.
  • Production experience with LLM systems such as RAG, agents, tool use, and evaluation, or demonstrated ability to ramp up quickly in these areas.
  • Strong testing and observability instincts, including reliable testing of nondeterministic components.
  • Working fluency with cloud infrastructure, containers, and CI/CD, with the ability to ship what you build.
  • Clear written and verbal communication with engineers, product leaders, and customers.
  • Eligibility for a US security clearance.

Nice To Haves

  • Prior principal- or staff-level experience on a platform used by other teams.
  • Experience scaling engineering practices such as code review, design standards, and onboarding as an organization grows substantially.
  • Experience with data platforms, feature stores, or ML pipelines.
  • Experience with optimization, simulation, or logistics systems.
  • Experience in constrained, air-gapped, or regulated deployments.
  • Background in military logistics, planning doctrine, or defense programs.

Responsibilities

  • Serve as a quality gate for AI services by setting standards for design, testing, and scalability and reviewing important changes before release.
  • Own the integration of AI services with the event-driven NATS-based backend, data models, and APIs in partnership with platform and backend engineers.
  • Review AI team code, including changes to shared backend and data systems.
  • Establish scalable review practices, including architecture guidelines, common patterns, design review templates, risk-based review tiers, and AI-assisted review.
  • Prevent architectural drift among engineers working in parallel, including those using AI coding assistants.
  • Set latency, throughput, concurrency, and cost targets; identify and address bottlenecks; and plan for substantially increased scale.
  • Write production code for difficult problems, including AI-specific backend features, agent and retrieval services, and data pipelines.
  • Lead design reviews and RFCs and drive them to decisions; plan safe infrastructure and data-model changes and rollouts.
  • Make build-versus-buy and model-selection decisions, including tradeoffs between self-hosted and cloud solutions under data-handling and deployment constraints.
  • Unblock the AI team's access to data sources, infrastructure, and tooling.
  • Mentor engineers, develop future technical leads, and help set the hiring bar.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service