AI Quality Engineer

Luma Financial Technologies • Cincinnati, OH

9d • $90,000 - $115,000

About The Position

Luma Fintech is building a best-in-class LLM-powered document parsing pipeline that extracts structured data from complex financial product term sheets. We are seeking an AI Quality Engineer to own the daily testing, analysis, and iterative improvement of our Claude API-based extraction system. This role sits at the intersection of financial data operations and applied AI, you will be the person who closes the loop between what the model outputs and what the schema demands.

Requirements

• Hands-on experience working with LLM APIs (Anthropic, OpenAI, or similar) in a production or near-production context
• Strong prompt engineering skills, you understand how instruction design affects model behavior, not just output tone
• Analytical mindset with the ability to systematically isolate variables in model output quality
• Experience designing structured test cases or evaluation frameworks (QA background is a plus)
• Familiarity with JSON schema, structured data output, and data validation patterns
• Ability to read and interpret complex financial or legal documents (term sheets, prospectuses, offering documents), prior financial services exposure strongly preferred
• Strong written communication; you’ll be documenting findings for both technical and non-technical stakeholders

Nice To Haves

• Experience with RAG pipelines and retrieval evaluation
• Python proficiency for scripting evaluation workflows or parsing outputs
• Background in structured financial products (autocallables, structured notes, credit-linked notes)
• Familiarity with evaluation frameworks or tools (e.g., LangSmith, RAGAS, custom evals)

Responsibilities

• Run daily accuracy evaluations against a defined extraction schema, tracking field-level performance across structured product types (autocallables, CLNs, barrier notes, etc.)
• Design and maintain test cases, regression suites, and gold-standard document sets to benchmark extraction quality over time
• Diagnose extraction failures, distinguishing between prompt logic issues, schema ambiguity, model hallucinations, and edge-case document formats
• Iterate on prompt engineering, system instructions, and context design to improve field-level extraction accuracy
• Work alongside the AI Engineer lead to feed findings into validation logic and rules-based layers that sit on top of LLM output
• Document failure modes with reproducible examples and root-cause hypotheses
• Build and maintain evaluation metrics (precision, recall, field coverage, hallucination rate) and report on accuracy trends
• Flag schema gaps or ambiguities surfaced by real document variance and collaborate with data operations to refine field definitions
• Contribute to RAG improvements by identifying where retrieved context is insufficient or misleading

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level
• Education Level: No Education Listed