Applied Data Scientist, Finance AI Evaluation & Datasets

Innodata Inc.

12d•CA$210,000 - CA$245,000

About The Position

Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence. We provide data, evaluation frameworks, and human expertise for building trustworthy AI systems. Our mission is to support Generative AI/AI builders and adopters with transferable solutions, platforms, and services, building on our 36+ year legacy of delivering high-quality data and outstanding customer outcomes. This role is crucial for the financial services domain, a high-stakes area for generative AI due to concerns like numerical accuracy, regulatory compliance, model risk management, auditability, and customer harm prevention. Innodata collaborates with foundation model labs, banks, asset managers, fintechs, and enterprise AI teams to build LLMs, multimodal systems, and AI agents for financial workflows. As an Applied Data Scientist, Financial AI Evaluation & Datasets, you will be responsible for the design, measurement quality, and domain validity of datasets used for training, fine-tuning, evaluating, and monitoring financial-domain LLMs, vision-language models, multimodal document models, and AI agents. This requires financial-domain fluency and data science rigor, enabling you to interpret financial documents, translate them into measurable dataset specifications, define key quality attributes (correctness, groundedness, compliance, safety), and produce trustworthy evidence for customers, model-risk teams, and AI governance stakeholders. The role emphasizes unstructured and multimodal financial data, including PDFs, scanned documents, spreadsheets, charts, call transcripts, and other mixed-document formats where text, numbers, visuals, and metadata are critical. You will work collaboratively in a pod with a Technical Solutions Architect, an Applied Research Scientist, an AI/ML Research Engineer, and Language Data Scientists to ensure domain validity, statistical defensibility, compliance, auditability, and usefulness of the team's outputs for evaluation and post-training.

Requirements

5+ years of data science experience, with at least 2+ years in financial services, fintech, banking, or a comparable regulated data environment.
Real working knowledge of financial data and workflows, including financial statements, SEC filings, transaction data, and other common financial-services document types.
Hands-on experience with unstructured and multimodal financial data (PDFs, scanned documents, spreadsheets, charts, or call transcripts).
Hands-on experience designing datasets for ML, including writing annotation guidelines, sizing cohorts, setting quality thresholds, and delivering data for training, evaluation, or monitoring.
Familiarity with LLM-based and multimodal financial AI workflows (prompt design, rubric-based evaluation, RAG, LLM-as-judge methods, and limitations of automated evaluation in high-stakes contexts).
Strong Python and SQL; comfort with pandas, scikit-learn, or equivalent; working familiarity with Hugging Face, PyTorch, or model APIs.
Statistical literacy, including sampling design, inter-annotator agreement metrics, confidence intervals, and critical evaluation of numerical interpretations.
Solid grasp of financial services privacy, compliance, and governance (PII handling, GLBA or equivalent privacy regimes, MNPI sensitivity, documentation for regulated AI programs).
Excellent collaboration skills with cross-functional teams.
A bias toward financial workflow realism, prioritizing datasets that reflect actual user scenarios.
Degree in a relevant field (statistics, data science, economics, finance, or related quantitative field) or equivalent demonstrated experience.
Experience designing evaluations for LLMs, VLMs, or multimodal models in financial reasoning, filings analysis, or fraud/AML contexts.
Experience with document AI, OCR/post-OCR quality, or table and chart extraction for complex financial documents.

Nice To Haves

Familiarity with financial standards or protocols such as XBRL, ISO 20022, or GAAP/IFRS reporting concepts.
Familiarity with agentic evaluation, AI observability, experiment tracking, or tools such as Weights & Biases or LangFuse.
Familiarity with model risk management frameworks, validation documentation, fairness/bias auditing, or consumer protection analysis.
Experience with multilingual or cross-border financial data, or published/open-source work in financial AI or model governance.
Formal finance credentials (CFA, FRM, MBA backgrounds) are especially encouraged.

Responsibilities

Translate customer goals into concrete dataset specifications, taxonomies, rubrics, and acceptance criteria for financial AI applications.
Design training and evaluation datasets across various financial AI surfaces, including financial QA, filings and earnings analysis, credit and underwriting, fraud/AML investigation, and compliance.
Prioritize unstructured and multimodal financial data (PDFs, scanned statements, tables, charts, call transcripts) in dataset design for analysts, advisors, compliance reviewers, and operations teams.
Design datasets and evaluations for retrieval-augmented and source-grounded systems, focusing on evidence citation, faithfulness to source documents, data freshness, conflict resolution, and failure modes from context parsing.
Evaluate agentic and workflow-integrated financial AI systems, assessing tool use, retrieval, transaction boundaries, escalation behavior, and safety controls.
Develop evaluation methodology that assesses numerical consistency, hallucination rates, refusal/escalation appropriateness, robustness under ambiguity, and fairness across customer segments.
Define sampling strategies, label schemas, and adjudication workflows with Language Data Scientists and finance SMEs, writing annotation guidelines for subjective judgments.
Build statistical and ML tooling for trustworthy financial datasets, including stratified sampling, bias analysis, leakage detection, and distribution shift checks.
Create evaluation and dataset-quality evidence for financial-services model risk management, including assumptions, limitations, validation results, and residual risks.
Partner with AI/ML Research Engineers to integrate datasets into training, evaluation, and monitoring pipelines using LLM-as-judge prompts, regression suites, and continuous monitoring.
Manage data quality end-to-end, including PII handling, provenance tracking, versioning, and modality-specific QA checks.
Reason about financial workflow context to understand AI output integration, reviewer needs, and uncertainty surfacing.
Support the Technical Solutions Architect in customer discovery and proposals by scoping dataset programs, sizing annotation effort, and explaining methodology.
Stay current on the financial AI landscape, including regulatory developments, benchmarks, and emerging evaluation methodologies.
Contribute to Innodata's internal IP, such as reusable taxonomies, evaluation rubrics, golden datasets, and methodology templates.