Senior Engineer, Enterprise AI Services
About The Position
Within Platform Engineering and Enterprise AI Services, the Senior Engineer, Enterprise AI Services is responsible for designing, building, and operating observability capabilities for AI and LLM workloads that power TR's AI-driven products. This role ensures our AI systems—from classical ML to generative AI—are observable, debuggable, reliable, and continuously improving across TR's multi-cloud footprint (AWS, Azure, GCP) and internal Kubernetes platform. You will own the end-to-end observability stack for AI (with Braintrust as foundational platform), enabling product teams, data scientists, and AI engineers to understand model behavior in production, detect issues early, and make data-driven improvements to model quality, latency, and cost. The successful candidate will help build the next generation of TR's AI observability and evaluation platform, working alongside cloud engineering, data engineering, Enterprise AI Services, and product teams.
Requirements
- Strong understanding of LLM/ML fundamentals and production AI systems, including prompting, context windows, RAG, hallucinations, and model/provider variability and capacity.
- Hands-on experience with AI observability and evaluation platforms, with Braintrust and/or Langfuse strongly preferred.
- Solid background with modern observability tooling, ideally Datadog.
- Deep Kubernetes experience deploying and operating services in production, including Helm-based releases, ingress/service networking, scaling, and troubleshooting.
- Proficiency in Python plus at least one additional backend or platform language such as TypeScript/JavaScript, Go, or Java. C# is an asset.
- Experience with API workloads across AWS, Azure, or GCP, including custom endpoints, and cloud-native production environments.
- 5+ years in SRE, Observability Engineering, ML Platform, or similar roles, including 2+ years supporting production LLM/ML systems, preferably in enterprise or regulated environments.
Nice To Haves
- Strong collaboration and communication skills, with the ability to work effectively across Product, Engineering, Data Science, SRE, and Cloud teams.
- Experience working in Agile environments, with familiarity in iterative delivery, sprint planning, backlog refinement, and cross-functional team execution.
- Demonstrated curiosity about AI technologies and a strong willingness to learn, adapt, and stay current with the rapidly evolving LLM/ML landscape.
- Strong problem-solving skills, sound judgment during incidents, and a continuous improvement mindset.
Responsibilities
- Serve as the Kubernetes expert for AI services, defining and operating deployment standards for scalability, resilience, security, and performance.
- Own the AI observability platform, implementing tools such as Braintrust and Langfuse to support tracing, evaluation, analytics, and monitoring of LLM/ML workloads.
- Define and standardize telemetry across AI products, including traces, metrics, logs, evaluations, and feedback, while ensuring governance, privacy, and auditability requirements are met.
- Build telemetry pipelines, dashboards, and reporting that provide clear visibility into model performance, quality, safety, reliability, and cost.
- Establish monitoring, alerting, SLOs/SLIs, and incident response practices for AI systems, including root cause analysis and continuous improvement.
- Integrate observability and evaluation into CI/CD so new models, prompts, and workflows are automatically enrolled in monitoring and quality controls.
- Partner with Product, Data Science, AI Engineering, SRE, Platform, and Cloud teams to onboard new AI use cases, support experimentation and drift detection, and implement guardrails and policy enforcement.
Benefits
- Hybrid Work Model
- Flexibility & Work-Life Balance
- Work from anywhere for up to 8 weeks per year
- Career Development and Growth
- Grow My Way programming
- Skills-first approach
- Flexible vacation
- Two company-wide Mental Health Days off
- Access to the Headspace app
- Retirement savings
- Tuition reimbursement
- Employee incentive programs
- Resources for mental, physical, and financial wellbeing
- Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more.
- Two paid volunteer days off annually
- Opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives.
- Market competitive health, dental, vision, disability, and life insurance programs
- Competitive 401k plan with company match
- Competitive vacation, sick and safe paid time off
- Paid holidays
- Parental leave
- Sabbatical leave
- Optional hospital, accident and sickness insurance paid 100% by the employee
- Optional life and AD&D insurance paid 100% by the employee
- Flexible Spending and Health Savings Accounts
- Fitness reimbursement
- Access to Employee Assistance Program
- Group Legal Identity Theft Protection benefit paid 100% by employee
- Access to 529 Plan
- Commuter benefits
- Adoption & Surrogacy Assistance
- Access to Employee Stock Purchase Plan
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
