Senior AI Engineer (Agentic Systems)

Pantheon Data • Reston, VA

2d • $120,000 - $220,000 • Remote

About The Position

Pantheon Data is seeking a Senior AI Engineer to design, build, and operate the agentic framework at the core of its AI platform in AWS GovCloud. The platform serves workloads that process CUI, PII, and PHI and must behave predictably, securely, and auditably under FedRAMP High and DoD IL4/IL5 requirements. The role owns agent runtime orchestration, routing, the capability registry, and the contracts through which capabilities plug in. It defines how capabilities are described, discovered, composed, and governed; machine learning and software engineers build capabilities such as retrieval, document extraction, and application integrations and expose them through the registry.

The Senior AI Engineer partners with the Data Engineer, who owns pipelines and data stores; the Cloud Security Engineer, who owns cloud security controls and ATO evidence; and DevSecOps and Cloud Operations Engineers, who own delivery and infrastructure. This is a systems engineering role focused on production agentic systems, not prompt-only applications or prototypes. Candidates should be able to discuss production systems they have built, including orchestration design, routing decisions and evaluation, failure containment, and possible improvements.

Pantheon Data is a private small business founded in 2011 and based in the Washington, DC, area. Its services include acquisition and supply chain management, infrastructure resiliency, contact center operations, information technology, software engineering, program management, strategic communications, engineering, and cybersecurity. Its customers include commercial clients and federal agencies, including DHS, DoD, and other Federal Civilian Agencies.

The role is listed as remote, with a work location of Reston, Virginia. Remote or hybrid employees may periodically work from a Pantheon Data office or client site when required by client needs. Local candidates should be prepared for an in-person interview; candidates outside the local area may be considered for a virtual interview. Applicants must be able to obtain and maintain a Secret clearance; an active Secret clearance is preferred. The salary range is $120,000–$220,000, with actual compensation depending on experience, geographic location, and potentially contractual requirements, and possibly outside that range.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related technical field from an accredited university, or equivalent professional experience.
• At least 8 years of professional software engineering experience, including at least 2 years building production LLM-backed systems.
• Demonstrated experience designing and shipping a production agentic system with multi-step or multi-agent orchestration and tool use, beyond single-prompt chat applications or demos.
• Hands-on experience with LLM tool calling, structured outputs, and at least one agent framework or SDK, such as LangGraph, Strands Agents, the Claude Agent SDK, Semantic Kernel, or a custom-built framework.
• Strong software architecture skills, including plugin- or registry-based designs, interface and schema design, versioning, and extensibility.
• Experience with distributed and asynchronous systems, including queues, event-driven workflows, concurrency, retries, idempotency, and failure handling.
• Experience evaluating nondeterministic systems, including building test sets, automated evaluations, and regression checks for LLM behavior.
• Strong Python engineering skills, including writing readable, maintainable, tested code; debugging; packaging; and API and service design.
• Working knowledge of AI security risks such as prompt injection, excessive tool permissions, and data exfiltration, and of controls that mitigate them.
• Experience with relevant AWS services, such as Amazon Bedrock, Lambda, Step Functions, ECS or EKS, SQS, DynamoDB, and CloudWatch.
• Experience with Git-based workflows, code review, CI/CD, and team-based software delivery.
• Clear written and verbal communication, including explaining agent behavior, tradeoffs, and failure modes to technical and non-technical stakeholders.
• Ability to work effectively remotely in cross-functional teams.
• Ability to meet deadlines and produce quality work.
• Proficiency in Microsoft Suite software, including Outlook, Word, Excel, SharePoint, and PowerPoint.
• Ability to obtain and maintain a Secret clearance.

Nice To Haves

• Experience building or operating agentic systems in AWS GovCloud or other environments supporting FedRAMP High or DoD IL4/IL5 workloads.
• Experience implementing or integrating MCP servers and clients, or agent-to-agent protocols, in production.
• Experience with Amazon Bedrock beyond basic inference, including Agents, Guardrails, Knowledge Bases, AgentCore, provisioned throughput, and cross-model routing.
• Experience building model routing or cascading strategies that balance quality, latency, and cost across multiple models.
• Experience with agent observability and tracing tools such as OpenTelemetry, Langfuse, LangSmith, or Arize Phoenix.
• Experience integrating retrieval, document processing, or other ML capabilities as tools within an agentic framework.
• Familiarity with federal compliance frameworks and their engineering implications, including NIST SP 800-53, CUI handling, audit logging, and least-privilege access in regulated environments.
• Experience with infrastructure as code (Terraform), containerized deployment, and GitLab CI or GitHub Actions in regulated environments.
• Advanced degree in a technical discipline, or demonstrated contribution to team technical direction and mentorship of other engineers.
• Active Secret clearance.

Responsibilities

• Design and build the orchestration layer to plan, dispatch, and coordinate multi-step and multi-agent workflows, including handoffs, parallel execution, cancellation, and human-in-the-loop checkpoints.
• Build routing that maps requests and intermediate steps to appropriate agents, tools, and models using suitable combinations of deterministic rules, classifiers, and LLM-based routing; measure routing accuracy and cost and make decisions observable and explainable.
• Own the capability registry, including capability schemas, versioning, discovery, access policy, and mechanisms to assemble agents at runtime from registered capabilities, tools, and instructions.
• Define tool and integration contracts, including tool definitions, Model Context Protocol (MCP) servers, input and output schemas, and error semantics, and help capability owners onboard to those contracts.
• Design agent state, session memory, and context management; implement retries, timeouts, idempotency, loop and budget limits, and graceful degradation to contain and recover from failures.
• Build evaluation harnesses for agent behavior, including routing accuracy, task completion, tool-call correctness, and regression testing across model and prompt versions.
• Trace agent runs end to end to support debugging and analysis of cost and latency.
• Enforce least-privilege tool access per agent and request, contain prompt injection across tool and agent boundaries, and record a complete audit trail of agent decisions and actions.
• Work with the Cloud Security Engineer to map security controls to NIST SP 800-53 and support ATO.
• Set architecture and standards for agentic development across the team, review designs and code, and mentor engineers building capabilities on the framework.

Benefits

• Competitive salaries and benefits.
• SmartBenefits through the Washington Metro Area Transportation Authority, allowing employees to direct a specified amount of pre-tax wages to a SmarTrip account.
• Tuition assistance may be available for continuing education expenses and certifications related to the position.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior