Principal Software Engineer, Applied AI
About The Position
Okta’s team that built Jeeva AI is developing Digital Workers, a suite of autonomous agents for identity operations and security. The team works directly with Okta’s co-founder and CEO and aims to bring the product to Okta’s 20,000+ customers. The role offers ownership and pace associated with a startup, with the resources of a public company. You will build and improve the harnesses and evaluations that make autonomous workers reliable as they investigate risk, use tools, wait for approvals, and complete tasks across enterprise systems. The position involves shaping product engineering direction across workflows and implementing shared components that help the team ship faster. The core stack includes Python FastAPI, AWS Step Functions, Amazon Bedrock AgentCore, PostgreSQL on Amazon RDS, and pgvector, with OpenAI and Anthropic model APIs. The team plans to add an agent harness and a dedicated analytics layer using ClickHouse or BigQuery; the harness choice is open.
Requirements
- Strong Python engineering skills and experience shipping an LLM application that uses tools.
- Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.
- Experience integrating model APIs, structured outputs, and asynchronous services; OpenAI or Anthropic experience is especially relevant.
- Depth in agent execution or evaluation systems, with working code and failure analysis that can be explained.
- Ability to distinguish a plausible model response from a supported finding or verified action.
Nice To Haves
- Experience with agent harnesses such as Claude Agent SDK, OpenAI Codex SDK, Strands Agents, LangChain Deep Agents, LangGraph, or a custom implementation; experience with equivalent tools also counts.
- Experience with agent observability, security evaluations, or synthetic evaluation data.
- Open-source contributions.
- Fine-tuning, preference data, post-training evaluation, or model distillation using approved datasets.
- Identity and security workflow experience.
- Foundation-model research experience is optional.
Responsibilities
- Build the agent harness for planning, tool use, memory, context compaction, interruption, and recovery during long-running work.
- Integrate OpenAI and Anthropic model APIs, select models based on measured quality, latency, and cost, and implement clear failure handling.
- Evaluate existing harnesses and build missing components while keeping the harness connected to AWS workflow execution and approval state.
- Keep authorization and sensitive writes behind application policy checks, and require evidence of resulting state before marking work complete.
- Build evaluation datasets and replay tests for valid findings, access decisions, app setup, and AI agent onboarding.
- Measure false positives, missed risks, unsafe actions, completion rate, and cost; turn production failures into regression tests.
- Test prompt injection, stale evidence, conflicting instructions, and unsafe tool use; protect tenant boundaries in retrieval and memory.
- Improve prompts, retrieval, tools, and model routing.
- Evaluate fine-tuning or other post-training methods when results justify them and data use is approved.
- Set product engineering direction across workflows and implement shared components that help the team ship faster.
Benefits
- Equity, where applicable.
- Bonus.
- Health insurance.
- Dental insurance.
- Vision insurance.
- 401(k).
- Flexible spending account.
- Paid leave, including PTO and parental leave.
- In-person onboarding experience.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
Education Level
No Education Listed
