Senior AI Engineer

Office of the State Auditor•Boston, MA

7d•$89,993 - $134,991

About The Position

The Senior AI Engineer designs, builds, and deploys AI-enabled applications in the Office of the State Auditor’s (OSA) Microsoft GCC environment. Working with the Innovation Assistant Director and Cloud Engineering, the role shapes OSA’s AI reference architecture and delivers production Retrieval-Augmented Generation (RAG) applications, agent workflows, supporting data pipelines, and cost- and performance-aware model deployments. The role uses existing foundation models through approved platforms such as Azure AI Foundry; it does not train or develop LLMs. The primary focus is delivering AI services for the Audit division that measurably improve audit planning, research, and analysis across OSA’s statewide audit portfolio. The role also builds AI-enabled programs for Human Resources, Administration & Finance, Occupational Schools, the Division of Local Mandates, and the Bureau of Special Investigations. The position owns technical design and engineering, collaborates with OSA teams and stakeholders, and does not supervise staff. It reports to the Innovation Assistant Director and works within OSA’s IT Innovation group.

Requirements

Bachelor’s degree in computer science, software engineering, or a related field, or equivalent experience.
5–7 years of professional software engineering experience, including at least 2 years building production applications using LLMs, such as RAG, agents, or tool-calling features.
Demonstrated ownership of the architecture for at least one production system, including documentation of design decisions.
Strong Python skills and working proficiency in TypeScript/JavaScript and SQL.
Hands-on experience building RAG systems with hybrid search and reranking, and experience with embeddings and vector stores.
Experience designing and deploying applications on Azure, or on AWS/GCP with willingness to work in Azure, including identity, private networking, and secrets management.
Experience with Docker, Kubernetes, infrastructure as code (Terraform or Bicep), and CI/CD pipelines.
Experience designing REST APIs and service boundaries, and writing automated tests.
Experience handling sensitive data (PII/PHI) under least-privilege access and encryption controls.
Clear written communication, including architecture documentation, and ability to explain technical trade-offs.

Nice To Haves

Azure Government/Microsoft GCC experience, including Azure Landing Zones and Cloud Adoption Framework implementation.
Microsoft Azure associate- or expert-level certifications, such as AI engineering or solutions architecture.
Experience with Microsoft Agent Framework and Prompt Flow migration; LangChain/LangGraph; structured outputs and function/tool calling across providers.
Deeper Kubernetes experience on AKS, including Helm charts, ingress, autoscaling, and workload identity.
Knowledge of knowledge graphs, taxonomies, or ontologies (such as RDF or SPARQL), and familiarity with Synaptica Graphite.
Data engineering with Databricks (Unity Catalog, Delta Lake) or Microsoft Fabric; data governance frameworks such as DMBOK; metadata and lineage management; and MDM.
RBAC/ABAC design, Purview classification, DLP labels, Defender for Cloud, and policy-as-code.
Understanding of LLM serving trade-offs, including quantization, KV-cache strategies, MoE models and expert parallelism, and open-weight serving stacks such as vLLM, TensorRT-LLM, or SGLang.
Experience in government, healthcare, or financial services, including shared platform or data standards across multiple agencies or business units.
Experience integrating TeamMate+ and external sources into AI-driven research workflows.
Experience mentoring developers or serving as a technical lead on a project.
Exposure to AWS or GCP data and AI services, such as Vertex AI.

Responsibilities

Design AI solution architectures, including components, integrations, data flows, security boundaries, and deployment topology; document decisions in diagrams and architecture decision records.
Lead definition and maintenance of OSA’s AI reference architecture, including patterns for RAG, agents, API exposure, logging, and evaluation, subject to review and approval by the Innovation Assistant Director.
Work with Cloud Engineering to extend Azure Landing Zone patterns for AI workloads, including subscriptions, Azure Policy, virtual networks, network security groups, private endpoints, Key Vault, and managed identities.
Define microservice and API boundaries so AI capabilities can be reused across audit and operational applications.
Lead technical design reviews and assess Azure AI services, vendor tools, and build-versus-buy options.
Design for cost, including model and hosting choices, budgets, and project-level usage tagging.
Develop, test, and maintain production Python and TypeScript/JavaScript code for AI applications, services, and interfaces.
Build REST APIs and backend services connecting LLMs to OSA documents, data sources, and business systems.
Write automated unit, integration, and evaluation tests; establish code review and Git/GitHub branching standards for AI projects.
Build automated tests, evaluation harnesses, and quality metrics for OSA’s software quality assurance process for AI applications.
Use AI-assisted development tools responsibly and review generated code for correctness and security.
Troubleshoot and improve AI components of production applications.
Own RAG retrieval design, including chunking, embedding choices, hybrid search, reranking, and citation fidelity checks, in collaboration with data engineering staff.
Engineer agent and tool-calling workflows with Microsoft Agent Framework and lead migration of Prompt Flow orchestration and evaluation into supported pathways.
Define and instrument AI quality and performance measures, and provide usage and adoption data for audit workflow KPIs and dashboards.
Build technical components of bounded pilots designed by the Manager, System and Process Improvements, and provide data to measure results against baselines.
Translate business requirements into technical requirements, prototypes, and production applications.
Containerize services with Docker and deploy them on Kubernetes (AKS), including health checks, autoscaling, secrets integration, and rolling updates.
Support the AI layer of deployed applications and transition hosting, infrastructure, and general application support to Enterprise IT through runbooks and knowledge transfer.
Build and maintain infrastructure as code and GitHub Actions CI/CD pipelines; design Dev/Test/Prod and sandbox patterns and promotion standards for AI assets.
Configure AI Foundry, APIM, Entra ID app registrations, and related Azure services for secure-by-default deployments.
Design backup, recovery, and business continuity approaches for AI services and document them in operational runbooks.
Design ingestion and transformation pipelines and establish patterns for data engineering staff preparing data for retrieval indexes, knowledge graphs, and analytics.
Define data interface contracts, including schemas, validation rules, and reconciliation checks, and build pipelines with retries, backfills, and idempotency.
Select and configure model deployments based on capability, latency, capacity, regional availability, and cost; evaluate provider, hosting, and accelerator options.
Apply application-level model optimizations, including caching, batch and streaming APIs, context and token budgeting, and model routing.
Integrate OSA taxonomy and entity relationships managed in Synaptica Graphite into retrieval and agent workflows, and expose graph-powered APIs for Audit Research Tool, Audit Search, and TeamMate+.
Model domain ontologies to support retrieval, entity linking, and policy-aware agent reasoning.
Design least-privilege access and implement OSA AI policies in code.
Implement audit-ready logging, FOIA-aware retention, and privacy controls such as DLP labels and redaction filters.
Build features supporting GAGAS-compatible documentation of AI assistance, human verification, and supervisory review.
Protect sensitive data through encryption, key management, and classification-driven access.
Partner with cybersecurity on threat modeling and security reviews, and maintain audit-ready evidence of controls.
Share technical knowledge through code reviews, documentation, and pairing on AI agent patterns, retrieval engineering, and reproducible deployments.
Build prioritized AI-enabled applications for OSA administrative and operational units, reusing shared platform components.
Provide day-to-day technical direction to the AI Application Developer and data engineering staff on application, pipeline, schema, and retrieval work.
Produce architecture documentation and operational runbooks that enable Enterprise IT and OSA staff to operate and extend systems without vendor dependency.
Explain architecture choices and trade-offs to the IT Innovation team and project stakeholders.