Senior AI Developer

ClickUp

1d • $200,000 - $250,000 • Remote

About The Position

ClickUp is seeking a Senior AI Engineer for its Engineering team, focused on AI evaluations. The role owns systems for measuring and improving the quality of ClickUp AI, including the evaluation systems, core agent harness, and AI capabilities that perform well against evaluations. It involves hands-on work with LLMs and backend systems to deliver scalable, reliable, and secure AI-powered features, solve user problems, and drive product innovation.

The position designs and operates evaluation infrastructure, builds agent self-improvement loops, benchmarks and routes models, and develops AI platform services and APIs. It also supports model serving, monitoring, logging, and automated evaluation in production. The role works across product, frontend, data, and engineering teams, and drives practices for AI privacy, security, and compliance.

ClickUp describes itself as an AI-native company and expects team members to use AI daily. AI fluency is part of the hiring process. The role is remote, as indicated by the job posting’s #LI-REMOTE tag. ClickUp is an equal opportunity employer. Engineering and product roles may be considered for visa sponsorship depending on business needs; sponsorship is not guaranteed.

Requirements

• At least 5 years of software engineering experience, including at least 2 years building production LLM or ML systems.
• Hands-on experience building production LLM evaluation systems, including golden datasets, LLM-as-judge, judge calibration, and offline and online evaluations tied to product quality.
• Strong experiment design and statistical rigor, including A/B testing, offline versus online metrics, and distinguishing real regressions from noisy or variable results.
• Proven experience applying LLMs and AI models to real-world, customer-facing product features.
• Experience integrating and managing multiple LLMs, including benchmarking, routing, and balancing cost, latency, and quality.
• Deep expertise in backend engineering, distributed systems, and cloud-native technologies such as Kubernetes, Docker, and AWS, GCP, or Azure.
• Strong programming skills in Python (preferred), Go, TypeScript, or similar languages used for backend and AI platform development.
• Proficiency in orchestration frameworks and workflow engines such as LangGraph, Airflow, Ray, or similar.
• Familiarity with search and retrieval technologies and their integration into AI-driven applications.
• Excellent collaboration and communication skills, with a record of effective cross-functional teamwork.
• Passion for staying at the forefront of AI and applying new technologies to real-world problems at scale.

Nice To Haves

• Background in data science, applied science, or machine learning engineering.
• Experience with evaluation tooling such as Braintrust, LangSmith, Weave, Arize, or in-house frameworks.
• Experience with model routing, AI gateways, or inference optimization.

Responsibilities

• Design, build, and own ClickUp’s AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features.
• Build and evolve the core agent harness and AI that performs well on evaluations, partnering with product and engineering teams to define quality goals.
• Build self-improvement loops in which agents modify other agents or prompts, rerun evaluations, and iterate automatically.
• Benchmark new models and own model selection, routing, and fallback strategies across providers, balancing quality, latency, and cost.
• Architect, design, and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs and other AI models.
• Build and maintain APIs and backend systems that integrate AI-powered features into ClickUp’s core platform.
• Develop infrastructure for model serving, monitoring, logging, and automated evaluation to support reliable, high-performing production AI services.
• Drive best practices for AI privacy, security, and compliance, including data anonymization, secure data handling, and regulatory adherence.
• Collaborate with product, frontend, and data teams to deliver seamless, reliable, user-centric AI experiences.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed