Senior AI Engineer

DoubleVerify • New York, NY

2d

About The Position

DoubleVerify is hiring a Senior AI Engineer for its Pinnacle AI (PinAI) team. The engineer will help build AgentOS, an agentic AI platform powering conversational experiences across the Pinnacle advertising analytics suite. The role involves developing AI agents, Model Context Protocol (MCP) servers, and LLM-backed tools so clients can explore advertising data using natural language. It is a hands-on engineering position focused on applied AI and production systems.

Requirements

• At least 5 years of professional software engineering experience building production-grade systems.
• At least 2 years of hands-on experience developing LLM-based applications or AI agent systems.
• Demonstrated experience shipping AI-powered features end to end, from prototype to production.
• Hands-on experience with LangChain, LangGraph, or equivalent agent orchestration frameworks.
• Familiarity with MCP or similar modular tool-integration protocols, or a strong appetite to ramp up quickly.
• Experience integrating LLM APIs, including streaming, tool-calling, and structured output patterns.
• Strong Python proficiency and ability to build clean, testable, production-quality backend services.
• Understanding of RAG patterns, vector stores, and semantic search.
• Experience building or using evaluation frameworks to measure LLM or agent quality and regression-test AI behavior.
• Familiarity with cloud environments such as GCP, AWS, or Azure, and containerization basics such as Docker or Kubernetes.
• Strong problem-solving skills and a bias toward shipping and iterating.
• Clear communication skills, including the ability to explain AI behavior and limitations to technical and non-technical stakeholders.
• Comfort operating with ambiguity in a fast-paced, evolving AI landscape.
• Collaborative approach and active contribution to shared standards and code quality.
• BS or MS in Computer Science, Machine Learning, AI, or a related technical field.

Nice To Haves

• Experience with MCP server development and tool-use patterns in production agentic systems.
• Familiarity with IAM, Keycloak, or OAuth 2.0-based authentication patterns.
• Knowledge of advertising analytics, AdTech platforms, or BI tools such as Looker or Snowflake.
• Experience with modern front-end frameworks such as React or Angular for conversational UI features.
• Contributions to open-source AI projects or published work in AI/ML.
• Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust.
• Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies.
• FastAPI experience.

Responsibilities

• Design, build, and maintain AI agents in the AgentOS framework using MCP, LangChain, LangGraph, and related tools.
• Develop MCP servers and tools that expose Pinnacle analytics capabilities, including metrics, dimensions, settings, and user data.
• Implement multi-turn conversational flows, tool-use reasoning chains, and agent orchestration for complex advertising analytics queries.
• Build and maintain the chat UI integration layer connecting AgentOS to the Pinnacle front end.
• Select, prompt, and integrate LLMs from providers such as OpenAI, Anthropic, and Google for specific agent tasks.
• Develop and own prompt engineering strategies, including system prompts, few-shot examples, and chain-of-thought techniques.
• Implement RAG pipelines grounded in Pinnacle proprietary data and documentation.
• Optimize model performance through evaluations, latency tuning, and cost management.
• Build and maintain evaluation frameworks measuring agent accuracy, instruction following, hallucination rates, and user satisfaction.
• Define quality benchmarks and run regression evaluations before production deployment of agent changes.
• Instrument agents with observability tools such as LangSmith, Weights & Biases, or custom tracing.
• Integrate AgentOS with Pinnacle platform services, including PIAM/Keycloak authentication, Settings APIs, and analytics data layers.
• Collaborate with backend teams to build and consume APIs that provide data securely to AI agents.
• Participate in architecture reviews and contribute to technical design documents for AI-powered features.
• Partner with Product, UX, and Data Science to translate business requirements into agent capabilities and conversational flows.
• Work with QE teams on testing strategies for AI features, including behavioral testing and adversarial prompting.
• Contribute to shared AI engineering standards, best practices, and documentation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior