Principal AI Engineer, GenAI Platform Operations

Johnson & Johnson Innovative Medicine•Raritan, NJ
•$105,000 - $181,700•Hybrid

About The Position

The Principal AI Engineer, GenAI Platform Operations is the senior technical authority for operating agents on Johnson & Johnson’s enterprise GenAI platform. The role sets the operations strategy and builds the capabilities needed to observe, manage and control agents in production across Microsoft Azure and AWS. Platform components include the MCP Gateway, agentic runtime, conversational AI and content-generation services, and Enterprise Knowledge Graph and GraphRAG retrieval services, supporting scientific, commercial, supply chain and manufacturing teams. This hands-on technical leadership role develops agent observability and performance telemetry, agent and dependency maps, automated reliability checks, and a runtime control plane able to throttle, pause, roll back or stop agents, tools and models. It owns agent lifecycle management at runtime, from registration and promotion through versioning, suspension and retirement, and leads the platform support and incident management teams. In partnership with Quality and Compliance and Information Security & Risk Management, the role ensures controls are validated, actions are traceable and the platform remains inspection-ready for regulated and patient-adjacent work. The position sets technical standards for support, site reliability, operations and platform engineers; establishes runtime readiness criteria for production agents; and uses agent performance signals to inform platform, agent, model, security and governance engineering priorities. It reports to the Senior Manager, TS GenAI platform engineering. The work pattern is hybrid, based in Raritan, New Jersey, and includes participation in escalation coverage for a 24x7 platform.

Requirements

  • At least 10 years of progressive technology experience, including at least 5 years in enterprise-scale services and 3 years of formal people leadership managing engineering, operations or support teams.
  • Bachelor’s degree or higher in Computer Science, Engineering, Information Systems or a related technical discipline; a master’s degree is preferred.
  • Demonstrated experience defining and technically delivering an operations, reliability or observability strategy and roadmap for a platform serving multiple internal consumer teams.
  • Track record designing and building observability platforms covering tracing, metrics, logs, alerting and service health reporting, and using observability data to drive engineering priorities.
  • Practical experience operating AI, machine learning or LLM-based services in production and monitoring their quality and performance, not only availability.
  • Experience building production runtime controls such as feature flags, circuit breakers, rate limiting, traffic management, emergency stops or automated rollback.
  • Hands-on operational experience with workloads on Microsoft Azure and/or AWS, including containerized services and managed data and networking services.
  • Experience operating technology in a regulated environment, with working knowledge of GxP change control, computer system validation and evidence expectations for audits or health authority inspections.
  • Experience automating operational work at scale, including health checks, triage and auto-remediation, and reducing toil through engineering.
  • Accountability for a 24x7 production service, including ITIL-aligned incident and problem management and an L1/L2 support function for a platform serving multiple internal consumer teams.
  • Experience building and leading support, incident management or reliability teams, including hiring, coaching, performance management and fostering an inclusive environment.
  • Ability to explain agent behavior, risks and control decisions clearly to senior Technology, business, Quality and governance stakeholders, including decisions to pause or stop an agent.
  • Willingness to participate in an escalation rotation supporting a 24x7 platform, including runtime control decisions, and to coordinate across global time zones during major incidents.
  • Knowledge of agent and LLM observability, including OpenTelemetry traces for multi-step agent runs and tool calls, and observability tools such as Azure Monitor, Application Insights, Amazon CloudWatch, Grafana, Datadog or Splunk.
  • Experience with production evaluation of LLM and agent quality, including online evaluation, LLM-as-judge scoring, groundedness and task-success metrics, and drift and regression detection.
  • Site reliability engineering practice, including SLI/SLO definition, error budget management, capacity planning, failure testing and toil reduction.
  • Incident command and major incident management, including severity models, escalation paths and executive communications.
  • ITIL-aligned incident, problem and change management and L1/L2 support operations, including hands-on experience with an enterprise ITSM platform such as ServiceNow.
  • Runtime control engineering, including kill switches, feature flags, circuit breakers, rate limiting, budget and quota enforcement, traffic shifting and automated rollback.
  • Agent lifecycle and release engineering, including registries, versioning, canary and shadow releases, promotion gates and retirement.
  • Experience building operations agents or AI-assisted automation with defined autonomy limits, human-in-the-loop approval and audit trails.
  • MCP Gateway and API gateway operations, including policy enforcement, authentication and tool-level access control.
  • Kubernetes and container operations on AKS and/or EKS, including scaling, upgrades, resource governance and production troubleshooting.
  • Operational depth in Azure and AWS platform services, including identity, networking, private connectivity, and quota and reserved capacity management.
  • Infrastructure as code and CI/CD for controlled, auditable releases using Terraform, GitHub Actions or Azure DevOps Pipelines.
  • Production-grade Python or Go skills for building operational tooling, automation and control services.
  • GxP change control and validation documentation practice, including traceable approvals and audit evidence for production controls and changes.
  • People leadership skills including hiring, onboarding, coaching, performance management, equitable workload and on-call design, and building an inclusive team consistent with Our Credo.
  • Contractor, vendor and managed service provider management.

Nice To Haves

  • Experience operating agentic or tool-calling workloads in production, including debugging multi-step agent executions and gateway-mediated MCP tool calls.
  • Familiarity with Arize, LangSmith and Langfuse.
  • Familiarity with agent runtimes and frameworks such as Amazon Bedrock AgentCore, Microsoft Foundry Agent Service, LangGraph or Strands Agents.
  • Experience designing an agent registry, catalog or inventory with ownership, risk tiering and dependency mapping.
  • Familiarity with AI risk and governance frameworks such as the NIST AI Risk Management Framework, ISO/IEC 42001 or the EU AI Act, and translating them into runtime controls.
  • Chaos engineering and resilience testing, including game days that exercise kill switches and failover.
  • FinOps experience for AI workloads, including token and inference cost monitoring, budget enforcement and cost per task.
  • Experience managing a service desk or L1/L2 function in partnership with a managed service provider.
  • Healthcare, pharmaceutical or medical device industry experience, with familiarity with 21 CFR Part 11, EU Annex 11 and Computer Software Assurance approaches.
  • Relevant certifications such as Azure or AWS professional-level certifications, Certified Kubernetes Administrator or ITIL 4.
  • Master’s degree in a related technical discipline.

Responsibilities

  • Define and own the GenAI platform operations strategy and roadmap for operating production agents across Azure and AWS, and publish agent operations reviews that translate performance, reliability, risk and cost trends into platform commitments.
  • Build the agent observability platform, including end-to-end traces for agent runs, MCP tool calls, model calls and retrieval using OpenTelemetry and generative AI semantic conventions; define the production instrumentation contract for agents.
  • Monitor production agent task completion, answer quality and groundedness, tool-call and MCP success rates, latency, token use, cost per task, guardrail interventions, and drift or regressions; establish SLOs and error budgets by agent and risk tier.
  • Create an authoritative agent registry, dependency maps and performance scorecards covering agent owners, versions, tools, MCP servers, models, data sources and risk tiers.
  • Design and deploy operations agents for health checks, synthetic probes, anomaly triage, evidence gathering and first-line remediation, with autonomy limits, human approval for consequential actions and audit trails.
  • Build and validate a runtime control plane with kill switches, circuit breakers, rate limits, budget caps, throttling, quarantine and gateway policy enforcement; coordinate validation under GxP change control with Quality and Compliance and ISRM, and maintain inspection-ready evidence of control actions.
  • Own runtime agent lifecycle management, including registration and admission, readiness gates, environment promotion, versioning, canary and shadow releases, automated rollback, suspension, deprecation and retirement.
  • Lead GenAI platform support operations, including L1/L2 intake, triage runbooks, routing and escalation to L3 engineering, response and resolution commitments, knowledge articles and known-error records; feed recurring issues and developer-facing gaps back to the responsible teams.
  • Lead incident management, including major incident command, severity classification, bridge coordination, stakeholder communications and service restoration; chair blameless reviews and problem management, and ensure corrective actions are completed.
  • Lead, hire, coach and develop support, incident management and reliability engineers; establish an inclusive and sustainable on-call rotation; manage cloud, observability and agent operations vendors and the managed service provider for L1 support.
  • Set technical standards and mentor site reliability, operations and platform engineers; define production readiness criteria for agents and turn performance signals into priorities for Azure and AWS platform engineering, agentic engineering, model engineering, and AI security and governance teams.
  • Participate in an escalation rotation for the 24x7 platform and coordinate across global time zones during major incidents.

Benefits

  • Medical insurance
  • Dental insurance
  • Vision insurance
  • Life insurance
  • Short- and long-term disability insurance
  • Business accident insurance
  • Group legal insurance
  • Pension plan
  • 401(k) savings plan
  • Vacation: 120 hours per calendar year
  • Sick time: 40 hours per calendar year, or 56 hours per calendar year for employees residing in Washington State
  • Holiday pay, including floating holidays: 13 days per calendar year
  • Work, Personal and Family Time: up to 40 hours per calendar year
  • Parental leave: 480 hours within one year of a child’s birth, adoption or foster care placement
  • Condolence leave: 30 days for an immediate family member and 5 days for an extended family member
  • Caregiver leave: 10 days
  • Volunteer leave: 4 days
  • Military spouse time off: 80 hours
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service