Principal AI Developer, oCTO Industry AI

SAP•Palo Alto, CA

2d•$198,200 - $420,000•Hybrid

About The Position

The Office of the CTO (oCTO) for IndustryAI sits at the intersection of the SAP Business AI Platform (BAIP), Industry Product & Engineering, the LoB application teams, and the FDE engagements with strategic customers. The Engineering team inside oCTO does not own product roadmaps for any single industry or LoB that lives with the product and engineering organizations. What we build is the AI-native and agentic software that turns the IndustryAI compounding loop into working systems. The pipelines that ingest customer data into ontologies and knowledge graphs, the harnesses that train and evaluate decision and foundation models, and the agents the IAI Data Labs and Agent Factory ship into production. If a customer trusts an agent to act on their operations, it is because someone built it to be correct, fast, and observable. As Principal AI Developer, you are the senior hands-on engineer who builds that software end-to-end. You own the design and implementation of complex AI-native and agentic systems within one or more capability areas. You set the engineering bar through the code you write and the reviews you run, and you turn ambiguous problems into shipped, production-grade capabilities. This is a deep individual-contributor role: you write the hardest code, you make the technical calls on one's own, and you are the engineer other engineers escalate to — while raising the craft of the developers around you.

Requirements

7–10 years of professional software-engineering experience, preferably in cloud or enterprise SaaS, with meaningful time building AI/ML or data-intensive systems.
Deep, current hands-on expertise building production software in AI-native systems — you write the code, and you are exemplary at it.
Demonstrated experience shipping enterprise-scale systems end-to-end. Distributed systems, cloud-native services, and ML or data platforms in production, not prototypes.
Practical experience building agentic and LLM-powered systems in production. Multi-agent orchestration, tool/function calling, RAG, context and memory engineering, and evaluation harness design.
Solid software-engineering fundamentals: clean design, testing discipline, performance and reliability instincts, and the judgment to know when to prototype fast and when to harden.
Fluency in cloud-native MLOps: CI/CD pipelines, model registries, feature stores, inference serving, observability stacks, and infrastructure as code.
Experience integrating models and agents against enterprise constraints — security, data governance, and compliance designed in from the start.
Comfort with specification-led engineering: the ability to decompose ambiguous problems, write precise specs, and hold the bar during implementation and review.
Exemplary technical communication: able to make design tradeoffs legible to architects, product, and fellow engineers alike.
Proficiency in one or more languages used for AI systems engineering (e.g., Python, TypeScript, Go, Java) and the modern AI/ML toolchain.
Bachelor's degree in Computer Science, Software Engineering, or a related discipline; advanced degree is a plus.

Nice To Haves

Domain fluency in one or more SAP-relevant areas (supply chain, manufacturing, asset management, procurement) is a plus.

Responsibilities

Design and build AI-native and agentic systems end-to-end. From data ingestion and knowledge-graph construction through model integration, agent orchestration, and production serving. As the senior implementing engineer, not just the reviewer.
Own the hardest engineering problems in your capability area: decompose ambiguous requirements into sound designs, prototype quickly, and drive them through to production-grade, well-tested code.
Build the agentic layer. Implement multi-agent orchestration, tool and function calling, retrieval-augmented generation (RAG), context, and memory engineering. As well as the guardrails that keep agents safe and reliable in production.
Integrate LLMs and ML models into enterprise systems — inference infrastructure, prompt and context pipelines, evaluation harnesses, and observability — grounded in what actually ships reliably at scale.
Practice specification-driven engineering. Write precise technical specs, validate designs against architectural guardrails. Hold the quality bar through implementation and code review.
Build the evaluation and observability that make AI systems credibility: eval suites, trace capture, regression harnesses, and the automated tests that catch drift and behavioral regressions before customers do.
Turn one-off solutions from FDE engagements into reusable, documented libraries and services that other IndustryAI engineering teams can adopt — code the whole organization builds on.
Feed the compounding loop from the engineering side: ensure production signal, traces, and incident learnings flow back into the harness, the eval suites, and the codebase so the systems get better between releases.
Contribute concrete requirements to BAIP. Orchestration primitives, memory and context APIs, inference tooling, evaluation harnesses, and observability hooks — grounded in what you need to build consistently.
Set the engineering standard through practice: CI/CD, MLOps, testing discipline, and responsible-AI practices embedded in how the team ships, not bolted on afterward.
Mentor senior and mid-level engineers on implementation, design tradeoffs, and code quality — multiplying your impact through the engineers you make better.
Partner with architects, data scientists, and product management to translate objectives into working systems with clear, measurable behavior.