Senior AI Engineer - Observability

OnBoard

1d•Remote

About The Position

OnBoard is a board intelligence platform trusted by over 6,000 organizations to simplify governance, and we are building AI-powered product experiences across RAG pipelines, semantic search, summarization, and emerging agentic workflows. We are looking for a Senior AI Engineer to help make those experiences reliable, measurable, cost-effective, and safe in production. This is a hands-on engineering role for someone who wants to build AI systems, not just observe them. You will partner with product engineering teams to design and improve AI features, instrument them deeply, and — critically — define how we know those features actually work. You will build the evaluation pipelines and datasets that tell us whether an AI experience is good enough to ship, analyze real-world behavior, and turn production signals into product and architecture improvements. You will help define how OnBoard ships AI: how we test prompts and retrieval quality, detect regressions, monitor cost and latency, evaluate user-facing quality, and safely evolve models, prompts, datasets, and providers over time. The right person is a strong software engineer with practical LLM application experience and a genuine quality mindset. You care not only that an AI feature works in a demo, but that it performs consistently for customers, degrades gracefully, provides traceable results, and improves through feedback loops. And you have real opinions about what makes an evaluation trustworthy — not just that one exists, but whether it measures the right thing.

Requirements

5+ years of software engineering experience building production systems
Hands-on experience building or operating LLM-powered features, RAG systems, AI workflows, or similar AI applications
Strong engineering ability in Python, C#/.NET, or both
Practical understanding of prompts, embeddings, vector search, retrieval quality, orchestration patterns, and LLM application architecture
Experience designing evaluations, quality metrics, or regression frameworks for AI or software systems — with the judgment to assess whether an evaluation actually measures what matters
Strong observability fundamentals: tracing, logging, metrics, alerting, and production debugging
Experience with CI/CD, git workflows, cloud environments, and production release practices (Azure DevOps preferred)
Ability to communicate clearly with engineering, product, QA, security, and business stakeholders
Strong product judgment and genuine curiosity about how AI systems behave with real users
Proficiency with AI-assisted development tools (e.g., Claude Code, PlayerZero)

Nice To Haves

Experience with LLMOps or AI observability tools such as Arize, Langfuse, LangSmith, W&B, Humanloop, or Helicone
Experience with OpenTelemetry, Azure Monitor, Application Insights, or similar observability platforms
Experience with Azure AI Search, Pinecone, Qdrant, Weaviate, pgvector, or other vector search platforms
Experience with Semantic Kernel, LangChain, LlamaIndex, AutoGen, or related frameworks
Experience with LLM-as-judge evaluation, RAG evaluation, semantic similarity metrics, hallucination detection, groundedness scoring, or human-feedback workflows
Experience with dedicated evaluation frameworks (e.g., DeepEval, LangTest) and benchmarking approaches for LLM outputs
Experience with A/B testing, online experimentation, or product analytics for AI features
Experience in regulated environments with SOC 2, ISO 27001, PII handling, or data-residency requirements
Background in QA, ML, or data engineering that informs a rigorous approach to quality measurement

Responsibilities

Build and improve AI-powered product systems
Partner with product engineers to design, build, and improve AI features using LLMs, RAG, semantic search, and agentic workflows
Contribute directly to production codebases in Python, C#/.NET, or related technologies
Improve prompt, retrieval, context assembly, ranking, grounding, and response-generation patterns
Help teams make practical architecture tradeoffs across quality, latency, cost, privacy, and maintainability
Support model and provider evaluations, migrations, fallback strategies, and rollout plans
Make AI quality measurable — define what "good enough" means
Design and implement evaluation pipelines for LLM-powered features, and build scoring methodologies from first principles rather than reaching for the nearest metric
Build and maintain versioned golden datasets covering real-world use cases, edge cases, failure modes, and customer-critical workflows
Implement LLM-as-judge, heuristic, human-feedback, and task-specific quality scoring approaches
Establish the criteria that determine whether an existing evaluation is sufficient for a given feature and risk profile — and identify gaps before they become production issues
Establish prompt and retrieval regression testing as part of the development lifecycle
Define quality gates and thresholds that help teams know when an AI feature is ready to ship
Own observability, reliability, and cost signals
Instrument LLM interactions, RAG pipelines, tool calls, and agent workflows using observability platforms (OnBoard currently uses Arize; comparable tools include Langfuse, LangSmith, W&B, and OpenTelemetry-based stacks)
Track latency, token usage, cost, retrieval quality, groundedness, failure modes, safety signals, and user feedback
Build dashboards and alerts that surface meaningful product and engineering signals, not just raw telemetry
Analyze production traces to identify quality issues, cost spikes, regressions, and improvement opportunities
Create runbooks and response patterns for common LLM and AI-product failure modes
Scale AI quality across teams
Build reusable libraries, SDKs, templates, and reference implementations that make correct AI instrumentation and evaluation easy
Document standards for tracing, metadata, prompt/version tracking, evaluation, cost reporting, and incident response
Coach product teams on AI quality, evaluation design, observability, and reliable release practices
Help establish shared patterns that let OnBoard scale AI development across product lines
Support responsible and compliant AI delivery
Ensure AI systems handle sensitive data appropriately and align with security, privacy, SOC 2, ISO 27001, and data-residency requirements
Monitor guardrails, policy enforcement, content safety signals, and safety-related anomalies as a distinct observability concern
Support auditability and traceability of AI interactions where required

Benefits

Fully remote work with company provided equipment (laptop, software, etc.)
Comprehensive, high-quality medical/prescription drug plan options, as well as dental and vision plan offerings.
An employer contribution to your Health Savings Account (HSA) if you participate in a High Deductible Healthcare Plan.
Medical Flexible Spending Accounts available.
Dependent Care Flexible Spending Accounts available.
Basic life insurance in the amount of $50,000 or 1 X’s your salary (whichever is higher).
Short and long-term disability and Accidental Death and Dismemberment benefits at no cost to you.
401K Retirement Savings Plan with automatic enrollment at the first of the month following 60 days of employment at 5% to help you secure your financial freedom. We offer a generous company match that starts on the first of the month following 60 days of employment. The company match is dollar for dollar on the first 3% of your pay that you contribute and $0.50 on the dollar on the next 2%, for a total match of 4%.
Paid Time Off (PTO)/Holiday
Employer paid Life and Accidental Death Insurance
Contribution to Health Care Spending Account
Dependent Life Insurance
Optional Life Insurance
LTD Insurance
Drug and Paramedical Coverage
Dental Insurance
Vision Insurance
EAP
Superannuation rate of 12%
Monthly stipend of $400 AUD to purchase private medical insurance
Pension - Aegon
Passageways/OnBoard contributes 8% of the employee's basic salary
Employees can contribute up to 100% of salary subject to max limits
Enrolled from Day 1 of employment
Private Medical Insurance
Life Assurance
Income Protection
Critical Illness
Employee Assistance Programme
Serious Illness Benefit
Help@Hand
Cashplan