BI Analyst/Report Developer

Codetech•Malvern, PA

About The Position

The BI Analyst/Report Developer validates and monitors generative AI tool outputs across structured, rules-based use cases and more complex AI-driven scenarios. The role uses trusted data sources, established SQL-based metrics, and scenario-based evaluation to assess output accuracy, consistency, explainability, reasonableness, and confidence. It also supports continuous improvement and responsible AI deployment by reviewing live interactions, identifying risks and discrepancies, and sharing findings with product, engineering, data engineering, and analytics teams.

Requirements

  • Robust SQL skills.
  • A robust analytical background with experience in data validation, SQL, and analytics workflows.
  • Ability to assess outputs quantitatively for data accuracy and qualitatively for reasonableness and business context.
  • Demonstrated critical thinking and sound judgment, especially in ambiguous or non-deterministic environments.
  • Experience working with large datasets, reporting tools, or analytics platforms.

Nice To Haves

  • Exposure to AI/ML or generative AI tools and associated testing or validation frameworks.
  • Experience in scenario-based testing, user acceptance testing (UAT), or model validation.
  • Familiarity with financial services, retirement, or plan sponsor analytics.

Responsibilities

  • Validate generative AI tool outputs for structured, rules-based use cases by reconciling results against trusted data sources and established SQL-based metrics.
  • Confirm outputs align with existing data pipelines and query logic to support consistency, explainability, and auditability.
  • Expand and maintain test coverage across prioritized use cases to establish a robust, high-confidence platform baseline.
  • Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic.
  • Design and execute scenario-based testing for complex AI-driven outputs where direct validation is not always possible.
  • Evaluate outputs for intent accuracy, reasonableness, and confidence thresholds rather than relying solely on exact-match validation.
  • Prioritize testing of high-risk and high-impact use cases using curated question sets and real-world scenarios.
  • Identify patterns in output variability and drive iterative refinement to improve reliability and user trust.
  • Review generative AI tool interactions after launch to validate outputs and ensure quality across user scenarios.
  • Identify edge cases, inconsistencies, and emerging risks, and escalate findings to product and engineering teams.
  • Synthesize testing and live-usage insights to inform enhancements, training data improvements, and governance practices.
  • Serve as an accountable reviewer and control point for responsible AI deployment and continuous improvement.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service