AI Transformation Engineer

Yext • New York, NY

1d • $142,500 - $213,500

About The Position

The AI Transformation Office partners with Yext departments on 30- to 60-day transformation sprints. The AI Transformation Engineer builds production systems using live data, including retrieval, answer drafting, routing, evaluation, APIs, automations, and pipelines. The role embeds with department teams to understand their processes, builds solutions using workflow platforms such as n8n where possible and code where needed, and transfers ownership to the team responsible for the outcome. The engineer builds systems that are operable by others and teaches partner teams as part of the work.

The position covers production LLM systems, evaluation and quality, knowledge transfer, system design, architecture, and identifying automation opportunities. Solutions are expected to be monitored, error-handled, version-controlled, evaluated against defined quality criteria, and designed with data permissions and sensitive information protections. Partner teams should be able to operate systems day to day within 30 days of launch, while the engineer retains retrieval, evaluation, and scoring code until another engineer can take it over.

Success is measured against agreed metrics and pre-launch baselines. Within 30 days, the engineer is expected to understand the partner department’s systems, identify at least three high-impact opportunities, and establish a metric, baseline, and evaluation set for the first sprint. Within 60 days, the first automation should be live and meet its quality threshold, and an embedded team member should be able to explain it and make minor changes. Within 90 days, the partner team should own the first system day to day, a second system should be live and stable, champions should be able to maintain and extend the systems, and team-level LLM spend and complete handoff documentation should be in place.

Requirements

• 5 to 8 years of experience in software engineering, integration development, or automation engineering.
• Production experience with a workflow platform such as n8n, Zapier, or Tray.io, building automations relied on by other teams.
• Experience designing reliable autonomous agents, including agent frameworks and patterns such as tool calling, orchestration, guardrails, human escalation, and evaluation feedback loops.
• Python or TypeScript skills for work that a platform cannot handle.
• API design skills and understanding of software operations, including monitoring, error handling, version control, and deployment discipline.
• Hands-on production experience with LLM APIs such as Anthropic, OpenAI, or Google, including prompt design and structured outputs.
• Retrieval experience, including vector databases such as Qdrant or pgvector, chunking, and grounding.
• Experience measuring LLM output quality with test sets, graded review, or evaluations.
• Experience handling sensitive data in production systems, including PII, permissions, and audit logging.
• Ability and inclination to teach, explain what is built and why, and recognize when someone needs help.
• Strong documentation discipline and ability to explain trade-offs to non-technical stakeholders without jargon.
• Ability to understand why a business process exists, contribute to prioritization, challenge assumptions about what is worth building, and work flexibly beyond a primary role in a small team.
• Comfort working in ambiguous environments where requirements evolve as understanding deepens.

Nice To Haves

• Experience with MCP servers, agent frameworks, or tool-calling design.
• Experience governing a shared LLM cost budget or API keys.
• Experience with CRM systems such as Salesforce or HubSpot, data engineering or ETL, and warehouse analytics such as Snowflake.
• Familiarity with support, customer success, or professional services operations.
• Experience building internal tools for colleague users.
• Training, facilitation, or mentoring experience, formal or informal.

Responsibilities

• Build and run production LLM workflows in n8n and similar platforms, including retrieval over ticket and documentation data, answer drafting, and routing.
• Build APIs, integrations, and data pipelines connecting CRM, support, and operational systems.
• Build and calibrate confidence scoring, and design human review steps and approval gates.
• Protect workflow data by scoping access, handling PII and sensitive fields, and maintaining access logs.
• Ship solutions with monitoring, error handling, and version control.
• Define evaluations before each build, including test sets, groundedness and accuracy checks, and thresholds for release or review.
• Report quality results to partner teams in clear, non-technical terms.
• Track LLM spend and usage by team, manage API keys, and report run rate.
• Pair-build with embedded team members so they learn through hands-on work.
• Create runbooks, admin guides, architecture diagrams, and decision logs.
• Train department champions to maintain and extend systems at an appropriate level.
• Design systems for handoff from the outset, enabling owning teams to manage workflow configuration, prompts, review queues, and reports day to day within 30 days of launch.
• Retain retrieval, evaluation, and scoring code until a second engineer can take it over.
• Design integration patterns and data flows that connect systems and enable automation and AI-powered decisions.
• Evaluate models and tools, and make build-versus-buy recommendations that account for the team's maintenance capacity.
• Identify automation and integration opportunities by understanding how work is performed.
• Translate technical possibilities into language department stakeholders can assess.
• Estimate effort realistically, scope tightly, and raise early when work will not affect the outcome.
• Agree on key engagement metrics, such as deflection rate, handle time, or hours saved, and measure results against pre-launch baselines.
• Build business context, contribute to prioritization, challenge assumptions about what is worth building, and develop business instincts.

Benefits

• Medical, dental, and vision benefits.
• Life insurance.
• Short-term and long-term disability coverage.
• 401(k) retirement plan.
• Vacation and sick leave.
• Potential eligibility for variable pay programs based on performance relative to goals and targets.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed