Principal Quality Engineer & AI Quality Archi

ELLKAY•Elmwood Park, NJ

17d•$110,000 - $115,000•Hybrid

About The Position

The Principal Quality Engineer and AI Quality Architect defines the technical vision, reference architecture, and engineering standards for quality across complex software, data, and AI-enabled products. This hands-on principal role architects scalable automation and evaluation platforms, governs AI-specific quality controls, and guides delivery teams through high-risk technical and regulatory decisions. The role owns enterprise quality engineering and AI assurance architecture, including principles, target-state capabilities, reference patterns, roadmaps, and measurable adoption standards. It designs modular automation and AI evaluation platforms, embeds quality controls into CI/CD, MLOps, and LLMOps workflows, and establishes standards for testability, environments, test data, telemetry, and quality metrics. The position also leads architecture reviews and proofs of concept, evaluates tools, and partners with security, privacy, legal, compliance, clinical, product, data science, and platform leaders to turn regulatory and responsible-AI obligations into verifiable controls. It coaches senior engineers and architects and influences cross-team technical and platform decisions.

Requirements

Bachelor's or master's degree in computer science, engineering, data science, AI, or a related discipline, or equivalent experience.
Typically 12+ years in software engineering or quality engineering, including enterprise architecture responsibility and recent leadership in AI or ML quality.
Expert-level coding and framework design skills in Python, Java, TypeScript, C#, or comparable languages, with strong API, distributed-system, cloud, CI/CD, and observability expertise.
Deep knowledge of test architecture, reliability engineering, security and privacy testing, data quality, performance engineering, and risk-based release governance.
Hands-on understanding of ML and generative-AI evaluation, RAG systems, embeddings or vector search, model monitoring, probabilistic behavior, and responsible-AI risks.
Ability to define enterprise standards while remaining hands-on with prototypes, reference implementations, design reviews, and complex technical investigations.

Nice To Haves

Experience architecting quality for regulated healthcare platforms and protecting PHI in AI-enabled workflows.
Familiarity with NIST AI RMF, model risk management, healthcare interoperability, and emerging AI governance or regulatory expectations.

Responsibilities

Own enterprise quality engineering and AI assurance architecture, including principles, target-state capabilities, reference patterns, roadmaps, and measurable adoption standards.
Architect modular automation platforms spanning UI, API, contract, integration, event, data, performance, resilience, security, accessibility, and production validation.
Define quality strategies for machine learning, generative AI, agentic workflows, retrieval-augmented generation, and predictive models across training, evaluation, deployment, and monitoring.
Establish AI evaluation frameworks covering task correctness, grounding, relevance, hallucination, robustness, bias and fairness, explainability, privacy, toxicity, safety, latency, and cost.
Design golden datasets, adversarial suites, synthetic test data, human-review protocols, statistical thresholds, and regression baselines for probabilistic systems.
Validate RAG retrieval quality, prompt and model changes, tool use, agent boundaries, fallback behavior, and defenses against prompt injection, data leakage, and unsafe actions.
Define model and prompt versioning, evaluation gates, lineage, audit evidence, drift detection, observability, rollback criteria, and incident-response patterns.
Embed automated quality controls into CI/CD, MLOps and LLMOps pipelines, cloud platforms, and developer workflows using fast feedback and policy-based release gates.
Set standards for testability, service virtualization, contract testing, environment strategy, test-data management, telemetry, and quality metrics across engineering portfolios.
Lead architecture reviews and proofs of concept; select tools through evidence-based evaluation of scalability, interoperability, security, maintainability, and total cost.
Partner with security, privacy, legal, compliance, clinical, product, data science, and platform leaders to translate regulatory and responsible-AI obligations into verifiable controls.
Coach senior engineers and architects, resolve cross-team technical constraints, and influence investment and platform decisions through clear tradeoff analysis.