Principal AI Engineer, Trust & Evaluation

Equinix•Toronto, ON
•$246,000 - $370,000

About The Position

The Principal AI Engineer, Trust & Evaluation sets the technical direction for the evaluation and trust capabilities of Equinix’s GenAI platform and designs, develops, and deploys those capabilities. The role builds platform features that let teams evaluate agents at scale, including golden datasets, synthetic test data, LLM-as-judge scoring, and production monitoring of agent quality. It also designs and tests trust controls for agents, including identity, access, guardrails, and data boundaries. As the hands-on technical lead of the Trust & Evaluation team, the engineer investigates difficult agent failures and partners with product, platform, security, legal, and governance leaders to turn ambiguous problems into robust, secure, observable, and performant solutions. The role applies expert software engineering practice and deep knowledge of LLM and agent concepts, evaluation methods and statistics, AI security risks, and identity and access concepts. It sets technical standards and practices for the team and other teams, keeping them current with industry research. The position addresses complex, open-ended, cross-team problems; sets quality metrics and release criteria; makes architecture decisions with long-term platform impact; and resolves trade-offs among quality, cost, latency, and security. The engineer partners with agent builders and forward-deployed engineers, translates their needs into a platform roadmap, communicates results and risks to senior and non-technical stakeholders, and mentors engineers. Success includes delivering the roadmap, enabling teams to evaluate agents without specialist help, maintaining auditable evaluation records for releases, enforcing tested trust controls, turning defects into regression tests, and spreading the team’s standards across Equinix.

Requirements

  • 7+ years in software, ML, or platform engineering, including 3+ years building ML or LLM systems in production.
  • Bachelor’s degree in computer science, engineering, data science, mathematics, or a related field, or equivalent experience.
  • Expert Python skills and solid version-control, testing, and code-review practices.
  • Track record of designing backend services or platform features used by many teams.
  • Track record of designing and leading the build of evaluation, testing, or quality platforms for ML or LLM applications, used by many teams or at scale.
  • Deep knowledge of LLM applications and agents, including prompting, retrieval, and tool calling, and how they fail; ability to trace difficult failures to root cause through logs and traces.
  • Deep knowledge of evaluation statistics, including sampling, confidence intervals, and variance across runs, and of measuring agreement between reviewers.
  • Ability to write SQL and analyze results.
  • Experience leading technical work across teams and influencing without direct authority.
  • Clear written and verbal communication, including with senior and non-technical stakeholders.

Nice To Haves

  • Experience with LLM evaluation and observability tools and tracing standards such as OpenTelemetry.
  • Experience with red-teaming or adversarial testing of LLM systems.
  • Experience with identity and access, including OAuth 2.0, OIDC, or delegated access.
  • Experience with performance, load, or test engineering.
  • Experience with agent frameworks, MCP, or A2A.
  • Experience with Google Cloud, including Vertex AI or GKE, and Kubernetes.
  • Familiarity with AI risk frameworks such as NIST AI RMF or ISO 42001.
  • Published research, patents, talks, or open-source contributions in evaluation or AI safety.
  • MS or PhD in a related field.

Responsibilities

  • Own the architecture and roadmap for platform evaluation features, and build dataset creation and versioning, golden datasets, synthetic test data, and reusable metric templates.
  • Design and build agent scoring approaches, including code-based checks, LLM-as-judge, tool-call accuracy, and grounding checks; determine which methods apply in different cases.
  • Build human-review features to compare judge scores with human reviewers and establish the agreement level required before teams rely on a judge.
  • Define how evaluations run before release, after changes to prompts, models, or tools, and for agents in production.
  • Define evaluation standards and playbooks for every agent release so builders can evaluate agents without specialist help.
  • Keep evaluation results traceable to the agent, model, prompt, and dataset versions tested, including records of release approvers for audit.
  • Design the platform’s trust architecture and lead its implementation, covering agent identity, per-user access to connected systems, tool permissions, guardrails, and data boundaries.
  • Lead adversarial testing and red-teaming, including prompt-injection, out-of-scope request, and data-boundary tests; turn findings into fixes and regression tests.
  • Partner with security and governance leaders to translate policy requirements into automated checks.
  • Lead end-to-end investigation of difficult agent failures and performance problems and follow them through to resolution.
  • Set the approach for test and performance tooling, including end-to-end agent tests, load and latency tests, and failure diagnostics.
  • Turn production failures and user feedback into regression tests to prevent recurring defects.
  • Write tested, reviewed, and documented code; contribute to shared libraries and tooling; and review team designs and code.
  • Ensure the team builds and tests to security, data-privacy, and responsible-AI requirements.
  • Partner with product, platform, security, legal, and governance leaders, as well as forward-deployed engineers and agent builders, to shape the platform roadmap.
  • Explain results, risks, and trade-offs to senior and non-technical stakeholders; mentor engineers and raise the team’s technical standards.

Benefits

  • Employee Assistance Program available to all employees.
  • Health or healthcare insurance coverage, with plans for eligible family members.
  • Life insurance.
  • Disability insurance.
  • Voluntary or optional benefit plans.
  • Retirement or savings plans, including US retirement plans and Canadian DCPP, RRSP, and TFSA options.
  • Accrued paid time off or vacation and personal time.
  • Paid holidays.
  • Bonus eligibility; equity may be offered depending on the position.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service