Software Architect

Vertex Inc.•Remote - PA, PA

About The Position

Serve as the technical authority for AI quality, evaluation, and risk controls within the Commercial AI Center of Excellence (CAI CoE). Define and operationalize technical mechanisms that ensure Vertex AI systems meet tax-grade standards for accuracy, defensibility, and compliance. Design and implement AI evaluation frameworks, including gold datasets, regression testing, and CI gates; define metrics across accuracy, confidence calibration, citation coverage, and abstention correctness. Lead model validation, error analysis, telemetry, and drift detection strategies; establish best practices for GenAI and ML hybrid systems. Partner with Legal, Security, and Risk teams to translate policy into enforceable technical mechanisms; review high-risk AI use cases and guide product teams on mitigation strategies. Influence standards and practices across large engineering organizations while partnering with product teams to improve model performance, reliability, and adoption.

Requirements

15+ years of experience in AI/ML engineering, with demonstrated impact at Senior Principal or equivalent level.
Deep hands-on experience designing and implementing AI/ML evaluation frameworks in production.
Expertise in model validation, drift detection, telemetry, and automated testing pipelines.
Strong understanding of AI governance, auditability, and risk controls.
Experience building evaluation harnesses, gold datasets, and CI-integrated regression testing.
Proven ability to partner cross-functionally with Product, Legal, Security, and Risk teams.
Strong system-level thinking in regulated SaaS environments.

Nice To Haves

Bachelor's degree in Computer Science, Engineering, or a related discipline; advanced degree preferred.
Industry experience in tax or other highly regulated domains (insurance, fintech, or healthcare).
Experience with vector databases and retrieval optimization.
FinOps for GenAI: experience modeling and optimizing LLM token and inference costs.
Data science or classical AI/ML background (beyond prompt engineering).
Strong executive communication and technical storytelling skills.
Equivalent combination of education, training, and relevant professional experience accepted in lieu of a formal degree.

Responsibilities

Serve as the technical authority for AI quality, evaluation, and risk controls within the Commercial AI Center of Excellence (CAI CoE).
Define and operationalize technical mechanisms that ensure Vertex AI systems meet tax-grade standards for accuracy, defensibility, and compliance.
Design and implement AI evaluation frameworks, including gold datasets, regression testing, and CI gates; define metrics across accuracy, confidence calibration, citation coverage, and abstention correctness.
Lead model validation, error analysis, telemetry, and drift detection strategies; establish best practices for GenAI and ML hybrid systems.
Partner with Legal, Security, and Risk teams to translate policy into enforceable technical mechanisms; review high-risk AI use cases and guide product teams on mitigation strategies.
Influence standards and practices across large engineering organizations while partnering with product teams to improve model performance, reliability, and adoption.