Applied Data Scientist, Finance AI Evaluation & Datasets

Innodata Inc.

3h•$150,000 - $175,000

About The Position

Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence. We provide data, evaluation frameworks, and human expertise for building trustworthy AI systems at scale, with a range of solutions for Generative AI builders and adopters. Our mission is to support the responsible advancement of AI by providing the necessary data, evaluation frameworks, and human expertise to build AI systems that can be trusted at scale. We offer a variety of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. With a 36+ year legacy, we are committed to delivering the highest quality data and outstanding outcomes for our customers. Financial services is a critical domain for generative AI, requiring stringent standards for numerical accuracy, regulatory compliance, model risk management, auditability, and customer harm prevention. Innodata collaborates with foundation model labs, banks, asset managers, fintechs, and enterprise AI teams to develop LLMs, multimodal systems, and AI agents for financial workflows. As an Applied Data Scientist, Financial AI Evaluation & Datasets, you will be responsible for the design, measurement quality, and domain validity of datasets used for training, fine-tuning, evaluating, and monitoring financial-domain LLMs, vision-language models, multimodal document models, and AI agents. This role demands financial domain fluency and data science rigor, enabling you to interpret financial documents, translate them into measurable datasets and evaluation specifications, define key metrics (correctness, groundedness, compliance, safety), and produce trustworthy evidence for sophisticated financial-services customers, model-risk teams, and AI governance stakeholders. This position has a strong focus on unstructured and multimodal financial data, including PDFs, scanned documents, spreadsheets, charts, call transcripts, and other mixed-document formats where text, numbers, visuals, and metadata are all crucial. You will collaborate within a pod structure, working alongside a Technical Solutions Architect, an Applied Research Scientist, an AI/ML Research Engineer, and Language Data Scientists to ensure that all outputs are domain-valid, statistically defensible, compliant, auditable, and suitable for evaluation and post-training purposes.

Requirements

5+ years of data science experience, with at least 2+ years in financial services, fintech, banking, or a comparable regulated data environment.
Real working knowledge of financial data and workflows: financial statements, SEC filings, transaction data, and other common financial-services document types.
Hands-on experience with unstructured and multimodal financial data — some combination of PDFs, scanned documents, spreadsheets, charts, or call transcripts.
Hands-on experience designing datasets for ML — not just consuming them. You have written annotation guidelines, sized cohorts, set quality thresholds, and shipped data that downstream teams could actually train, evaluate, or monitor on.
Familiarity with LLM-based and multimodal financial AI workflows: prompt design, rubric-based evaluation, RAG, LLM-as-judge methods, and the limitations of automated evaluation in high-stakes contexts.
Strong Python and SQL; comfort with pandas, scikit-learn, or equivalent; working familiarity with Hugging Face, PyTorch, or model APIs.
Statistical literacy: sampling design, inter-annotator agreement metrics (e.g., Cohen's kappa), confidence intervals, and the ability to push back when a number is being over-interpreted.
Solid grasp of financial services privacy, compliance, and governance: PII handling, GLBA or equivalent privacy regimes, MNPI sensitivity, and documentation fit for regulated AI programs.
Excellent collaboration skills — upstream with a Technical Solutions Architect, sideways with research scientists and engineers, and downstream with SME annotators and quality teams.
A bias toward financial workflow realism. You would rather build a smaller dataset that reflects what analysts, advisors, or customers actually see than a larger one that looks impressive on paper but fails in practice.
Degree in a relevant field — statistics, data science, economics, finance, or a related quantitative field, or equivalent demonstrated experience. Formal finance credentials aren't required, but CFA, FRM, or MBA backgrounds, etc. are especially encouraged.

Nice To Haves

Familiarity with financial standards or protocols such as XBRL, ISO 20022, or GAAP/IFRS reporting concepts, etc. is strongly preferred.
Experience designing evaluations for LLMs, VLMs, or multimodal models in financial reasoning, filings analysis, or fraud/AML contexts.
Experience with document AI, OCR/post-OCR quality, or table and chart extraction for complex financial documents.
Familiarity with agentic evaluation, AI observability, experiment tracking, or tools such as Weights & Biases or LangFuse.
Familiarity with model risk management frameworks, validation documentation, fairness/bias auditing, or consumer protection analysis.
Experience with multilingual or cross-border financial data, or published/open-source work in financial AI or model governance.

Responsibilities

Translate customer goals — such as improving financial reasoning, building an eval suite for earnings-call summarization, or evaluating an AML/fraud copilot — into concrete dataset specifications, taxonomies, rubrics, and acceptance criteria.
Design training and evaluation datasets across the financial AI surface: financial QA, filings and earnings analysis, credit and underwriting, fraud/AML investigation, and compliance, among other financial workflows.
Foreground unstructured and multimodal financial data in dataset design — PDFs, scanned statements, tables, charts, and call transcripts — used by analysts, advisors, compliance reviewers, and operations teams.
Design datasets and evaluations for retrieval-augmented and source-grounded systems: evidence citation and faithfulness to source documents, data freshness, conflict resolution across sources, and failure modes caused by incomplete or incorrectly parsed context.
Evaluate agentic and workflow-integrated financial AI systems: tool use, retrieval, transaction boundaries, escalation behavior, and controls that prevent unsafe or unauthorized actions.
Develop evaluation methodology that goes beyond surface accuracy — numerical consistency, hallucination rates on high-risk claims, refusal and escalation appropriateness, robustness under ambiguity, and fairness across protected or sensitive customer segments.
Define sampling strategies, label schemas, and adjudication workflows with Language Data Scientists and finance SMEs; write annotation guidelines that make subjective finance-domain judgments explicit, calibratable, and auditable.
Build the statistical and ML tooling that makes large financial datasets trustworthy: stratified sampling across products, markets, and modalities; bias analysis; leakage detection; and distribution shift checks, among other reliability checks.
Build evaluation and dataset-quality evidence to support financial-services model risk management: assumptions, limitations, validation results, and residual risks, packaged as reproducible evidence.
Partner with the AI/ML Research Engineer to instrument datasets into training, evaluation, and monitoring pipelines — rubric-grounded LLM-as-judge prompts, regression suites, and continuous monitoring.
Own data quality end-to-end, from intake through delivery: PII handling, provenance tracking, versioning, and modality-specific QA checks.
Reason about financial workflow context: where AI outputs enter analyst, advisor, compliance, risk, or customer-facing workflows; what evidence a reviewer needs to trust them; and when uncertainty must be surfaced.
Support the Technical Solutions Architect during customer discovery and proposals: scoping dataset programs, sizing annotation effort, and explaining methodology to client stakeholders.
Stay current on the financial AI landscape: regulatory developments, benchmark releases, and emerging evaluation methodology for finance-domain models.
Contribute to Innodata internal IP: reusable taxonomies, evaluation rubrics, golden datasets, and methodology templates.