Senior Software Engineer, Applied AI

Okta•San Francisco, CA

About The Position

Okta is building Digital Workers, a suite of autonomous agents for identity operations and security. The role will help develop the harnesses and evaluations that make these agents reliable as they investigate risk, use tools, wait for approvals, and complete tasks across enterprise systems. The team works directly with Okta co-founder and CEO Todd McKinnon, and the product is intended to reach Okta’s 20,000+ customers. The role combines startup-like ownership and pace with the resources of a public company. The engineer will build and improve agent execution and evaluation capabilities, integrate model APIs, and make evidence-based decisions about model quality, latency, and cost. The work includes connecting the harness to AWS workflow execution and approval state, enforcing application policy checks for authorization and sensitive writes, and requiring evidence of resulting state before marking work complete. The engineer will also develop evaluation datasets and replay tests, investigate production failures, and improve prompts, retrieval, tools, and model routing. Fine-tuning and other post-training methods may be evaluated when justified by results and approved data use. The core stack includes Python FastAPI, AWS Step Functions, Amazon Bedrock AgentCore, PostgreSQL on Amazon RDS, and pgvector. OpenAI and Anthropic model APIs are part of the integration plan. Okta plans to add an agent harness and a dedicated analytics layer using ClickHouse or BigQuery; the harness choice is open, and equivalent tools are accepted. The Senior-level scope is to own an agent capability or evaluation pipeline and deliver measurable improvements.

Requirements

Strong Python engineering skills.
Experience shipping an LLM application that uses tools.
Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.
Experience integrating model APIs, structured outputs, and asynchronous services.
Depth in agent execution or evaluation systems, with working code and failure analysis that can be explained.
Ability to distinguish a plausible model response from a supported finding or verified action.

Nice To Haves

Experience with Claude Agent SDK, OpenAI Codex SDK, Strands Agents, LangChain Deep Agents, LangGraph, or a custom agent harness implementation.
Experience with OpenAI or Anthropic model APIs.
Equivalent experience with the listed stack and tools.
Open-source contributions.
Agent observability experience.
Security evaluations experience.
Synthetic evaluation data experience.
Fine-tuning, preference data, post-training evaluation, or model distillation using approved datasets.
Experience with identity and security workflows.
Foundation-model research experience (optional).

Responsibilities

Build the agent harness for planning, tool use, memory, context compaction, interruption, and recovery during long-running work.
Integrate OpenAI and Anthropic model APIs; select models based on measured quality, latency, and cost, with clear failure handling.
Evaluate existing harnesses and build only the missing capabilities.
Connect the harness to AWS workflow execution and approval state.
Keep authorization and sensitive writes behind application policy checks.
Require evidence of resulting state before marking work complete.
Build evaluation datasets and replay tests for valid findings, access decisions, app setup, and AI agent onboarding.
Measure false positives, missed risks, unsafe actions, completion rate, and cost.
Turn production failures into regression tests.
Test prompt injection, stale evidence, conflicting instructions, and unsafe tool use.
Protect tenant boundaries in retrieval and memory.
Improve prompts, retrieval, tools, and model routing.
Evaluate fine-tuning or other post-training methods when results justify them and data use is approved.
Own an agent capability or evaluation pipeline with measurable improvements at the Senior level.