Senior AI Engineer (Agentic Systems)

Pantheon Data • Reston, VA

3d • $120,000 - $220,000 • Remote

About The Position

Pantheon Data is seeking a Senior AI Engineer to design, build, and operate the agentic framework at the core of its AI platform in AWS GovCloud, supporting workloads that process CUI, PII, and PHI. The role owns the agent runtime, including orchestration, routing, the capability registry, and the contracts that capabilities use. It is responsible for making agent systems predictable, secure, and auditable under FedRAMP High and DoD IL4/IL5 requirements.

The engineer defines how capabilities such as retrieval, document extraction, and application integrations are described, discovered, composed, and governed. They partner with data engineering, cloud security, DevSecOps, and cloud operations colleagues. This is a production systems engineering role requiring experience building and operating agentic systems, rather than a prompt-only or prototype role.

The position is remote, with a work location listed as Reston, Virginia. Remote or hybrid employees may periodically work at a Pantheon Data office or client site for specific requirements. Local candidates should be prepared for an in-person interview; candidates outside the local area may be considered for a virtual interview. Candidates must be able to obtain and maintain a Secret clearance; an active Secret clearance is preferred. The salary range is $120,000–$220,000, with actual compensation based on experience, geographic location, and potentially contractual requirements.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related technical field from an accredited university, or equivalent professional experience.
• At least 8 years of professional software engineering experience, including at least 2 years building production LLM-backed systems.
• Demonstrated experience designing and shipping a production agentic system with multi-step or multi-agent orchestration and tool use, beyond single-prompt chat applications or demos.
• Hands-on experience with LLM tool calling, structured outputs, and at least one agent framework or SDK, such as LangGraph, Strands Agents, the Claude Agent SDK, Semantic Kernel, or a custom framework.
• Strong software architecture skills, including plugin- or registry-based designs, interface and schema design, versioning, and extensibility.
• Experience with distributed and asynchronous systems, including queues, event-driven workflows, concurrency, retries, idempotency, and failure handling.
• Experience evaluating nondeterministic systems, including building test sets, automated evaluations, and regression checks for LLM behavior.
• Strong Python engineering skills, including readable, maintainable, tested code; debugging; packaging; and API and service design.
• Working knowledge of AI security risks such as prompt injection, excessive tool permissions, and data exfiltration, and controls that mitigate them.
• Experience with relevant AWS services such as Amazon Bedrock, Lambda, Step Functions, ECS or EKS, SQS, DynamoDB, and CloudWatch.
• Experience with Git-based workflows, code review, CI/CD, and team-based software delivery.
• Clear written and verbal communication, including explaining agent behavior, tradeoffs, and failure modes to technical and non-technical stakeholders.
• Ability to work effectively remotely in cross-functional teams.
• Ability to meet deadlines and produce quality work.
• Proficiency in Microsoft Suite software, including Outlook, Word, Excel, SharePoint, and PowerPoint.
• Ability to obtain and maintain a Secret clearance.

Nice To Haves

• Experience building or operating agentic systems in AWS GovCloud or other environments supporting FedRAMP High or DoD IL4/IL5 workloads.
• Experience implementing or integrating MCP servers and clients, or agent-to-agent protocols, in production.
• Experience with Amazon Bedrock beyond basic inference, including Agents, Guardrails, Knowledge Bases, AgentCore, provisioned throughput, and cross-model routing.
• Experience building model routing or cascading strategies that balance quality, latency, and cost across multiple models.
• Experience with agent observability and tracing tools such as OpenTelemetry, Langfuse, LangSmith, or Arize Phoenix.
• Experience integrating retrieval, document processing, or other ML capabilities as tools within an agentic framework.
• Familiarity with federal compliance frameworks and their engineering implications, including NIST SP 800-53, CUI handling, audit logging, and least-privilege access in regulated environments.
• Experience with infrastructure as code (Terraform), containerized deployment, and GitLab CI or GitHub Actions in regulated environments.
• Advanced degree in a technical discipline, or demonstrated contribution to team technical direction and mentorship of other engineers.
• Active Secret clearance.

Responsibilities

• Design and build the orchestration layer for multi-step and multi-agent workflows, including handoffs, parallel execution, cancellation, and human-in-the-loop checkpoints.
• Build request and intermediate-step routing to agents, tools, and models using appropriate deterministic rules, classifiers, and LLM-based routing; measure routing accuracy and cost and make decisions observable and explainable.
• Own the capability registry, including capability schemas, versioning, discovery, access policy, and runtime assembly of agents from registered capabilities, tools, and instructions.
• Define tool and integration contracts, including tool definitions, Model Context Protocol (MCP) servers, input and output schemas, and error semantics; help capability owners onboard.
• Design agent state, session memory, and context management; implement retries, timeouts, idempotency, loop and budget limits, and graceful degradation.
• Build evaluation harnesses for routing accuracy, task completion, tool-call correctness, and regression across model and prompt versions; trace agent runs for debugging, cost, and latency.
• Enforce least-privilege tool access, contain prompt injection across tool and agent boundaries, and record audit trails of agent decisions and actions; work with the Cloud Security Engineer to map controls to NIST SP 800-53 and support ATO.
• Set architecture and standards for agentic development, review designs and code, and mentor engineers building capabilities on the framework.

Benefits

• Competitive salaries and benefits.
• SmartBenefits through the Washington Metro Area Transportation Authority, with an amount of pre-tax wages paid directly to a SmarTrip account.
• In some cases, tuition assistance may be available for continuing education expenses and certifications related to the position.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior