Senior AI Engineer

DoubleVerify•New York, NY
•Hybrid

About The Position

DoubleVerify is seeking a Senior AI Engineer to join its Pinnacle AI (PinAI) team and build AgentOS, an agentic AI platform powering conversational experiences across the Pinnacle advertising analytics suite. The role involves designing and implementing AI agents, Model Context Protocol (MCP) servers, and LLM-backed tools that let clients interact with advertising data using natural language. The engineer will develop agent workflows and integrations, implement retrieval-augmented generation (RAG), and work with major LLM providers. They will also establish evaluation and observability practices, integrate AgentOS with existing platform services, and collaborate with Product, UX, Data Science, backend, and QE teams. The position is based in New York, NY, with hybrid work flexibility.

Requirements

  • At least 5 years of professional software engineering experience building production-grade systems.
  • At least 2 years of hands-on experience developing LLM-based applications or AI agent systems.
  • Demonstrated experience shipping AI-powered features end-to-end, from prototype to production.
  • Hands-on experience with LangChain, LangGraph, or equivalent agent orchestration frameworks.
  • Familiarity with MCP or similar modular tool-integration protocols, or a strong appetite to ramp up quickly.
  • Experience integrating LLM APIs, including streaming, tool-calling, and structured output patterns.
  • Strong Python proficiency and ability to build clean, testable, production-quality backend services.
  • Understanding of RAG patterns, vector stores, and semantic search.
  • Experience building or using evaluation frameworks to measure LLM or agent quality and regression-test behavior.
  • Familiarity with cloud environments such as GCP, AWS, or Azure, and containerization basics such as Docker or Kubernetes.
  • Strong problem-solving skills and a bias toward shipping and iterating.
  • Clear communication skills, including explaining AI behavior and limitations to technical and non-technical stakeholders.
  • Comfort operating with ambiguity in a fast-paced, evolving AI landscape.
  • Collaborative approach and active contribution to shared standards and code quality.
  • BS or MS in Computer Science, Machine Learning, AI, or a related technical field.

Nice To Haves

  • Experience developing MCP servers and production agentic-system tool-use patterns.
  • Familiarity with PIAM, Keycloak, or OAuth 2.0-based authentication patterns.
  • Knowledge of advertising analytics, AdTech platforms, or BI tools such as Looker or Snowflake.
  • Experience with modern front-end frameworks such as React or Angular for conversational UI features.
  • Contributions to open-source AI projects or published work in AI/ML.
  • Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust.
  • Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies.
  • FastAPI experience.

Responsibilities

  • Design, build, and maintain AI agents within the AgentOS framework using MCP, LangChain, LangGraph, and related tools.
  • Develop MCP servers and tools that expose Pinnacle analytics capabilities, including metrics, dimensions, settings, and user data.
  • Implement multi-turn conversational flows, tool-use reasoning chains, and agent orchestration for advertising analytics queries.
  • Build and maintain the chat UI integration layer connecting AgentOS to the Pinnacle front end.
  • Select, prompt, and integrate LLMs from providers such as OpenAI, Anthropic, and Google.
  • Develop prompt engineering strategies, including system prompts, few-shot examples, and chain-of-thought techniques.
  • Implement RAG pipelines to ground AI responses in Pinnacle's proprietary data and documentation.
  • Optimize model performance through evaluations, latency tuning, and cost management.
  • Build and maintain evaluation frameworks for agent accuracy, instruction-following, hallucination rates, and user satisfaction.
  • Define quality benchmarks and run regression evaluations before production deployment.
  • Instrument agents with observability tools to support debugging and continuous improvement.
  • Integrate AgentOS with platform services, including PIAM/Keycloak authentication, Settings APIs, and analytics data layers.
  • Collaborate with backend teams to build and consume APIs that provide data securely to AI agents.
  • Participate in architecture reviews and contribute to technical design documents for AI-powered features.
  • Partner with Product, UX, and Data Science to translate business requirements into agent capabilities and conversational flows.
  • Work with QE teams to define AI feature testing strategies, including behavioral testing and adversarial prompting.
  • Contribute to shared AI engineering standards, best practices, and documentation.

Benefits

  • Competitive compensation including base salary, bonus, and equity.
  • Comprehensive benefits package.
  • Professional development and conference opportunities.
  • Hybrid work flexibility from the New York City headquarters.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service