Principal Software Engineer, Applied AI

Okta•San Francisco, CA

21h

About The Position

Okta is building Digital Workers, a suite of autonomous agents for identity operations and security. The team, which built Jeeva AI, works directly with Okta’s co-founder and CEO and aims to bring the product to Okta’s 20,000+ customers. This role helps shape the product with startup-like ownership and pace, supported by the resources of a public company. The engineer will build the harnesses and evaluations that enable agents to investigate risk, use tools, wait for approvals, and complete tasks across enterprise systems reliably. The Principal Software Engineer will set product engineering direction across workflows and implement shared components that help the team ship faster. The role focuses on agent execution and model quality, with ownership spanning code, evaluation, release, and customer use. The core stack includes Python FastAPI, AWS Step Functions, Amazon Bedrock AgentCore, PostgreSQL on Amazon RDS, and pgvector. OpenAI and Anthropic model APIs are part of the integration plan, and the team plans to add an agent harness and a dedicated analytics layer using ClickHouse or BigQuery.

Requirements

Strong Python engineering skills and experience shipping an LLM application that uses tools.
Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.
Relevant agent harness experience, such as Claude Agent SDK, OpenAI Codex SDK, Strands Agents, LangChain Deep Agents, LangGraph, or a custom implementation.
Experience integrating model APIs, structured outputs, and asynchronous services.
Depth in agent execution or evaluation systems, with working code and failure analysis that can be explained.
Ability to distinguish a plausible model response from a supported finding or verified action.

Nice To Haves

Experience with OpenAI or Anthropic model APIs is especially relevant.
Experience with equivalent tools to those in the listed stack.
Open-source contributions.
Agent observability experience.
Security evaluations or synthetic evaluation data experience.
Fine-tuning, preference data, post-training evaluation, or model distillation using approved datasets.
Identity and security workflow experience.
Foundation-model research experience is optional.

Responsibilities

Build the agent harness for planning, tool use, memory, context compaction, interruption, and recovery during long-running work.
Integrate OpenAI and Anthropic model APIs and select models based on measured quality, latency, and cost, with clear failure handling.
Evaluate existing agent harnesses and build only missing components.
Connect the harness to AWS workflow execution and approval state.
Keep authorization and sensitive writes behind application policy checks.
Require evidence of resulting state before marking work complete.
Build evaluation datasets and replay tests for valid findings, access decisions, app setup, and AI agent onboarding.
Measure false positives, missed risks, unsafe actions, completion rate, and cost.
Convert production failures into regression tests.
Test prompt injection, stale evidence, conflicting instructions, and unsafe tool use.
Protect tenant boundaries in retrieval and memory.
Improve prompts, retrieval, tools, and model routing.
Evaluate fine-tuning or other post-training methods when results justify them and data use is approved.
Set product engineering direction across workflows and implement shared components that help the team ship faster.