Forward Deployed Engineer - Assistant Vice President

Citi • New York, NY

9d • $121,200 - $181,800

About The Position

The Applications Development Senior Programmer Analyst — Forward Deployed Engineer (C12) is an intermediate-level software engineering position embedded in line-of-business delivery pods. The role builds automated evaluation harnesses, curates regression testbeds, optimizes prompts and output schemas, and automates testing pipelines to ensure the reliability, accuracy, and safety of production AI agents. Evaluation suites are built in Agentflow Studio, with automated CI/CD test execution.

Requirements

• 5–8 years of relevant software engineering or systems analysis experience, or 2–4 years with a specialized focus in Python and generative AI application development.
• Proficiency in Python, with practical experience integrating REST or gRPC APIs, Docker containers, and Git workflows.
• Working knowledge of AI evaluation frameworks, benchmarking methodologies such as BLEU, ROUGE, semantic similarity, and LLM-as-a-judge, and automated testing suites.
• Experience managing and implementing automated test pipelines in CI/CD environments such as Lightspeed, Tekton, or Jenkins.
• Ability to work effectively under pressure while managing tight sprint deadlines and unexpected changes in model outputs or line-of-business requirements.
• Good written and verbal communication skills and a methodical, detail-oriented approach to software quality and data integrity.
• Bachelor’s or university degree, or equivalent experience, in Computer Science, Data Science, or a related quantitative discipline.

Responsibilities

• Construct, calibrate, and execute automated evaluation test suites using Agentflow Studio to measure agent task completion, tool-call correctness, latency, and hallucination metrics.
• Curate and maintain domain-specific golden datasets, synthetic benchmarks, and regression testbeds in collaboration with line-of-business domain experts and senior engineers.
• Optimize prompt templates, system instructions, few-shot examples, and structured JSON output parsers using Pydantic or JSON Schema to improve deterministic agent behavior.
• Author and maintain automated test execution and deployment pipelines within Lightspeed CI/CD, enforcing continuous evaluation gates before staging and production rollout.
• Use Python application-development expertise to analyze complex agent errors, evaluate system performance, and recommend coding and prompt solutions.
• Recommend and implement security measures and Lakera Guardrail policies following implementation analysis of agent usage.
• Monitor and control evaluation test cycles, provide operational support and performance metrics, and prepare prompt diagnostic reports for business stakeholders and squad leads.
• Operate with limited direct supervision, exercising independent judgment and autonomy while acting as an evaluation subject matter expert for pod team members.
• Assess risk in business decisions, protect the firm's reputation and the assets of Citigroup and its clients, comply with applicable laws and policies, exercise sound ethical judgment, and transparently escalate, manage, and report control issues.

Benefits

• Medical, dental, and vision coverage
• 401(k)
• Life, accident, and disability insurance
• Wellness programs
• Paid time off, including vacation, sick leave, and paid holidays
• Eligible employees may receive discretionary and formulaic incentive and retention awards

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level