Principal Production Engineer

Paramount•New York, NY

About The Position

The Principal Production Engineer is the senior technical authority for the Integration Team’s delivery: architecting and shipping stakeholder-facing AI workflow solutions that are reliable, observable, secure, and maintainable. This role is a hands-on solver who tackles the hardest integration problems, defines solution-layer reference architectures and standards, and ensures multiple concurrent engagements execute with consistent technical quality. They bridge product intent and engineering execution by translating ambiguous workflow needs into scalable technical designs and production-ready implementations.

Requirements

10+ years in tools/integration engineering or equivalent senior-level expertise demonstrated through portfolio-impact technical management and shipped production systems.
Expert Python (primary). Demonstrated ability to lead integrations across heterogeneous stacks (e.g., Go/TypeScript/C++) strongly preferred.
Secondary languages may be used for web front-ends, small services, plugins, or integrating into existing studio pipelines.
Proven ability to ship and operate web apps/services with production-grade observability and reliability.
Deep experience with APIs/service integration, auth/SSO, and data systems (including vector/retrieval systems) in constrained enterprise environments.
Strong cross-functional communication and influence skills; able to drive platform and research partners toward concrete interface improvements.

Nice To Haves

Production experience with AI/ML or LLM-based tools, including eval/quality loops and safety considerations.
Familiarity with MCPs or similar orchestration/tooling frameworks.
Media, entertainment, or production workflow background.
Experience with security/privacy/compliance-heavy environments (on-prem, regulated data, strict network egress).

Responsibilities

Own reference architecture for the solution layer across engagements (boundaries, data flows, error/fallback behavior, safe UX patterns, operational controls). Lead the design for complex integrations. This involves working with several systems, limited environments, and secure VPC or on-prem setups. You will also conduct threat modeling and apply privacy-by-design when necessary.
Take point on the hardest integration work: performance bottlenecks, reliability gaps, deployment constraints, data quality edge cases, and cross-system failure modes.
Build or guide critical-path components and reference implementations; create reusable libraries where leverage is high.
Define and evolve reference patterns for delivery shapes (API/MCP, plugin, automation, batch, embedded service, secure enterprise patterns).
Ensure that platform modules follow the rules for interfaces. This includes validating schemas, pinning versions, checking compatibility, mapping dependencies, and managing changes.
Set the production definition of done. This includes telemetry, dashboards, alerts, runbooks, rollback plans, and supportability. Conduct readiness reviews for high-impact releases.
Run architecture reviews and targeted technical audits to reduce integration churn and regressions.
Partner with Core AI/Research & Platform to improve module usability and integration fit (latency budgets, failure modes, data formatting, eval needs).
Identify recurring integration issues. Propose improvements for the platform. These improvements should aim to reduce the need for custom solutions. This can involve using shared components, observability hooks, and standardized contracts.
Plan for cross-brand compatibility when the platform changes. Create playbooks for upgrades and deprecations. These playbooks will help protect existing solutions.
Standardize reliability patterns across the portfolio. This includes rate limiting and backoff strategies. It also includes retries, circuit breakers, caching, idempotency, and graceful degradation.
Ensure end-to-end instrumentation standards are met (usage, success/failure, latency, cost proxies, workflow KPIs).
Lead production hardening on critical engagements. This means validating stability, participating in incidents, and making improvements based on postmortem reviews. Follow through on ongoing actions.
Ensure operability artifacts exist and are usable (dashboards, alerts, runbooks, escalation paths).
Ensure the quality and consistency of the Solution Bundle. This means maintaining architecture notes. It also includes managing integration patterns, operational learnings, reusable components, and known issues.
Build and maintain shared scaffolding (templates, SDK wrappers, test harnesses, telemetry middleware) to accelerate repeat engagements.
Find ways to create standardized integration packages from successful patterns. Also, think about related capabilities of the platform.
Cross-brand reference architectures, standards, and golden path integration patterns.
Exceptions to standards and significant architectural departures.
Contract changes for platforms and modules include schemas and API/MCP behavior. We also have requests for the platform guidelines. We need to manage multi-pod dependencies due to technical reasons.
Portfolio-level production readiness bar and changes to go/no-go criteria and rubrics.
Strategies for upgrading or removing systems. It also covers compatibility approaches that impact several pods and brands.
Pod-level sequencing and sprint-ready scope (with PM partnership).
Solution design and implementation within established standards.
Release readiness for pod-owned solutions, provided readiness criteria are met.
First-line escalation and incident triage for pod-owned systems.
Coherent, maintainable architecture and standards across the integration portfolio.
Success of complex/critical integrations and technical risk reduction across brands.
Production readiness bar adopted and met consistently (telemetry, reliability, operability).
Platform interface improvements driven by integration feedback (reduced bespoke work over time).
Reusable artifacts delivered (Solution Bundles, shared components) that measurably increase leverage.
Fewer integration regressions and reduced churn from platform changes; improved compatibility and upgrade success.
Higher pilot-to-production conversion; fewer late-stage surprises due to environment or security constraints.
Dashboards and runbooks consistently in place; fewer repeat incidents; faster resolution on integration-owned issues.
Measurable workflow KPI improvements and robust stakeholder feedback.
Increased reuse of shared patterns/components; fewer bespoke one-off implementations across engagements.