Senior AI Engineer

Commonwealth of Massachusetts•Boston, MA

7d•$89,993 - $134,991

About The Position

The Senior AI Engineer designs, builds, and deploys AI-enabled applications in the Office of the State Auditor’s Microsoft GCC environment. Working with the Innovation Assistant Director and Cloud Engineering, the role shapes OSA’s AI reference architecture and delivers production Retrieval-Augmented Generation (RAG) applications, agent workflows, supporting data pipelines, and cost- and performance-aware model deployments. The role uses existing foundation models through approved platforms such as Azure AI Foundry; it does not train or develop large language models. Its primary focus is delivering AI services that improve audit planning, research, and analysis across OSA’s statewide audit portfolio. It also develops AI-enabled programs for Human Resources, Administration & Finance, Occupational Schools, the Division of Local Mandates, and the Bureau of Special Investigations. The position owns the technical design and engineering of solutions and does not supervise staff. It reports to the Innovation Assistant Director within OSA’s IT Innovation group and collaborates with Cloud Engineering, Enterprise IT, the Data Analytics Unit, data engineering staff, contractors, and business stakeholders. The role provides day-to-day technical direction to the AI Application Developer and data engineering staff. It designs AI solution architectures; develops and tests production software; engineers retrieval, agent, and model workflows; builds deployment and operational systems; prepares data for AI applications; implements security and governance controls; and documents systems so OSA staff and Enterprise IT can operate and extend them. The position is full-time by default under the supplied employment-type instruction. Salary is $89,993.11 to $134,991.01 per year, commensurate with experience.

Requirements

Bachelor’s degree in computer science, software engineering, or a related field, or equivalent experience.
5–7 years of professional software engineering experience, including at least 2 years building production applications that use LLMs, such as RAG, agents, or tool-calling features.
Demonstrated ownership of the architecture for at least one production system, including documentation of design decisions.
Strong Python skills and working proficiency in TypeScript/JavaScript and SQL.
Hands-on experience building RAG systems with hybrid search and reranking, and experience with embeddings and vector stores.
Experience designing and deploying applications on Azure, or on AWS/GCP with willingness to work in Azure, including identity, private networking, and secrets management.
Experience with Docker, Kubernetes, infrastructure as code (Terraform or Bicep), and CI/CD pipelines.
Experience designing REST APIs and service boundaries and writing automated tests.
Experience handling sensitive data such as PII/PHI under least-privilege access and encryption controls.
Clear written communication, including architecture documentation, and ability to explain technical trade-offs.

Nice To Haves

Azure Government/Microsoft GCC experience, including Azure Landing Zones and Cloud Adoption Framework implementation.
Microsoft Azure associate- or expert-level certifications, such as AI engineering or solutions architecture.
Experience with Microsoft Agent Framework and Prompt Flow migration; LangChain/LangGraph; structured outputs; and function/tool calling across providers.
Deeper Kubernetes experience on AKS, including Helm charts, ingress, autoscaling, and workload identity.
Knowledge of knowledge graphs, taxonomies, or ontologies, such as RDF or SPARQL; familiarity with Synaptica Graphite.
Data engineering experience with Databricks (Unity Catalog, Delta Lake) or Microsoft Fabric; data governance frameworks such as DMBOK; metadata and lineage management; and MDM.
Knowledge of RBAC/ABAC design, Purview classification, DLP labels, Defender for Cloud, and policy as code.
Understanding of LLM serving trade-offs, including quantization, KV-cache strategies, MoE models and expert parallelism, and open-weight serving stacks such as vLLM, TensorRT-LLM, or SGLang.
Experience in government, healthcare, or financial services, including shared platforms or data standards across multiple agencies or business units.
Experience integrating TeamMate+ and external sources into AI-driven research workflows.
Experience mentoring developers or serving as a project technical lead.
Exposure to AWS or GCP data and AI services, such as Vertex AI.

Responsibilities

Design AI application architectures, integrations, data flows, security boundaries, and deployment topology; document designs with diagrams and architecture decision records.
Lead the definition and maintenance of OSA’s AI reference architecture, including standard patterns for RAG, agents, API exposure, logging, and evaluation, subject to review and approval by the Innovation Assistant Director.
Work with Cloud Engineering to extend Azure Landing Zone patterns for AI workloads, including subscriptions, Azure Policy, virtual networks and network security groups, private endpoints, Key Vault, and managed identities.
Define microservice and REST/API Management boundaries so AI capabilities can be reused across audit and operational applications.
Lead technical design reviews and assess Azure AI services, vendor tools, and build-versus-buy options to inform technology decisions.
Design AI deployments for cost visibility and control, including model and hosting choices, budgets, and project-level tagging.
Write, test, and maintain production Python and TypeScript/JavaScript code for AI applications, services, and user interfaces.
Build REST APIs and backend services connecting language models to OSA documents, data sources, and business systems.
Write automated unit, integration, and evaluation tests; establish code review and Git/GitHub branching standards for AI projects.
Build automated tests, evaluation harnesses, and quality metrics for OSA’s software quality assurance process for AI applications.
Use AI-assisted development tools responsibly and review generated code for correctness and security.
Troubleshoot and improve AI components of applications in production.
Own RAG retrieval design, including chunking, embedding choices, hybrid search, reranking, and citation-fidelity checks, in collaboration with data engineering staff.
Engineer agent and tool-calling workflows using Microsoft Agent Framework and lead migration of existing Prompt Flow orchestration and evaluation into supported pathways.
Define and instrument AI quality and performance measures, including groundedness, relevance, citation accuracy, latency, throughput, concurrency, and cost per query; create monitoring dashboards and provide usage and adoption data for audit workflow KPIs.
Build technical components of bounded pilots designed by the Manager, System and Process Improvements and provide data to measure results against baselines.
Translate business requirements into technical requirements, prototypes, and production applications, working with the Manager, System and Process Improvements for Audit or the Innovation Assistant Director for other units.
Containerize services with Docker and deploy them on Kubernetes/AKS, including health checks, autoscaling, secrets integration, and rolling updates.
Support the AI layer of deployed applications and transition hosting, infrastructure, and general application support to Enterprise IT through runbooks and knowledge transfer.
Build and maintain infrastructure as code using Terraform or Bicep and GitHub Actions CI/CD pipelines; design Dev, Test, Prod, and sandbox environments and promotion standards for AI services and assets.
Configure Azure AI Foundry, API Management, Entra ID app registrations, and related Azure services for secure-by-default deployments.
Design backup, recovery, and business continuity approaches for AI services and document them in runbooks.
Design ingestion and transformation pipelines and establish patterns for data engineering staff to prepare audit, agency, and internal operational data for retrieval indexes, knowledge graphs, and analytics.
Define data interface contracts, schemas, validation rules, and reconciliation checks; build pipelines with retries, backfills, and idempotency.
Select and configure model deployments based on capability, latency, capacity, regional availability in GCC, and cost.
Evaluate model provider and hosting options, including accelerator options, for throughput, latency, and cost.
Apply application-level optimizations such as prompt and prefix caching, batch and streaming APIs, context and token budgeting, and routing between models.
Integrate OSA’s taxonomy and entity relationships, managed in Synaptica Graphite, into retrieval and agent workflows; expose graph-powered APIs for Audit Research Tool, Audit Search, and TeamMate+.
Model domain ontologies, including entities, events, and regulations, to support retrieval, entity linking, and policy-aware reasoning in agents.
Design least-privilege access using Entra ID security-group patterns, RBAC, and PIM; implement OSA AI policies as code.
Implement audit-ready logging, FOIA-aware retention, and privacy controls such as DLP labels and redaction filters.
Build application features supporting GAGAS-compatible documentation of AI assistance, human verification, and supervisory review, as defined with the Manager, System and Process Improvements and Quality Assurance.
Protect sensitive data, including PII, PHI, and HIPAA-regulated healthcare data, with encryption, key management, and classification-driven access.
Partner with cybersecurity on threat modeling and security reviews and maintain audit-ready evidence of controls.
Share technical knowledge through code reviews, documentation, and pairing on agent patterns, retrieval engineering, and reproducible deployments.
Build shared-platform AI applications for OSA operational units as prioritized by the Innovation Assistant Director.
Provide day-to-day technical direction to the AI Application Developer and data engineering staff on application, pipeline, schema, and retrieval work.
Produce architecture documentation and operational runbooks that enable Enterprise IT and OSA staff to operate and extend systems without vendor dependency.
Explain architecture choices and trade-offs to the IT Innovation team and project stakeholders.