Software Engineer, AI Platform

Aalyria•San Francisco, CA

2d•$185,000 - $215,000•Hybrid

About The Position

Aalyria is hiring a software engineer to build the AI layer of our engineering organization. This is fundamentally a building role where you will design, ship, and operate internal AI products and platform services, such as chat and agent interfaces, retrieval over our code and docs, AI-integrated developer workflows, sandboxed execution environments, on top of our existing cloud infrastructure (GCP, GKE, Vertex AI, GitLab). We operate in a regulated environment (CMMC, DoD Impact Levels, FedRAMP), and that shapes the work. Your job is to work with our security and compliance director, ask the right questions, extract the real constraints, and then engineer systems that deliver the maximum utility to our engineers while staying inside the boundary. The integrations you build need care and feeding: they must keep working, keep improving, and remain auditable. You will own them end to end.

Requirements

Several years building, shipping, and operating production software.
Fluency in at least one of Python, Go, or TypeScript, plus Terraform.
Comfort with API design, testing, code review, and owning services in production.
Hands-on experience building LLM-powered systems: retrieval pipelines, tool use / MCP, agent orchestration, prompt and context management, and evaluating whether any of it actually works.
Product sense for internal tooling: experience deploying, extending, and integrating open-source applications (chat frontends, gateways, dev tools) rather than building everything from scratch, with a strong instinct for build vs. configure vs. wait.
Developer-platform integration experience: working with Git-hosting APIs, webhooks, and CI/CD to embed AI into the workflows engineers already live in.
Cloud and Kubernetes engineering: you build and operate the infrastructure your systems run on yourself, such as containers, Helm, Terraform, GKE, and/or IAM.
Sandboxing and isolation literacy: understanding of how to safely run model-generated code and constrain agent access (containers, network policy, least-privilege credentials).
Effective in a regulated environment: not compliance expertise, but the ability to elicit constraints from security stakeholders, ask the right questions, design within hard boundaries, and build systems whose behavior is observable and auditable by construction.
Clear written communication: Specifically for technical and executive audiences, including candid assessments of what is not working.
High autonomy: you will define your own roadmap from a clear mandate and validate it directly with the teams you serve.

Nice To Haves

Prior work in CMMC, DoD IL, FedRAMP, or NIST 800-171 environments — especially running AI/LLM workloads inside such a boundary.
Self-hosted inference experience: vLLM/TGI/similar, GPU provisioning and utilization, open-weight model deployment.
Experience with LLM gateway/proxy layers (e.g., LiteLLM or equivalent) and per-team cost attribution.
Hardware-in-the-loop or lab-automation experience: test benches, device access control, or safely bridging software systems to physical equipment.
Familiarity with DLP concepts as they apply at the model/API layer.
GCP specifically (Vertex AI, GKE, IAP, Artifact Registry); Bazel or other hermetic build systems.
Observability stack experience (OpenTelemetry, Grafana/Loki/Mimir/Tempo or similar).

Responsibilities

Deploy and extend an open-source frontend (e.g., LibreChat, Open WebUI) backed by our existing Vertex AI models and any future inference infrastructure, with SSO, access policy, and the integrations that make it actually useful, RAG and MCP servers tied to our GitLab repos and internal docs, code execution (Code Interpreter-style), and internal tool access.
Develop agents that generate merge requests, respond to review comments, and iterate, so engineers can drive AI work entirely through the review interface they already use.
Create isolated, policy-enforced environments where agents (and the engineers supervising them) can safely run code, access repos, and use tools — extending to infrastructure that lets AI safely interact with real hardware in our lab environments.
Implement request-level telemetry, usage analytics, and cost attribution across all model usage, so we know how AI is being used internally and what it's worth.
Identify, build, and maintain LLM-powered systems across engineering and operations workflows, prioritizing leverage over coverage.
Design and build agentic systems and internal AI applications: multi-step pipelines, tool-using agents, retrieval-augmented systems, and internal-facing apps that put model capabilities in front of the right people.
Own the AI gateway layer: model routing, credential management, access policy, and request-level telemetry across all model usage.
Operate what you ship: monitoring, upgrades, incident response, and continuous improvement of the AI stack.
Partner with the security/compliance director to understand boundary requirements (CUI handling, data residency, provider selection, audit logging) and translate them into system design, building guardrails in at design time, not as afterthoughts.
Contribute the technical substance (architecture, data-flow diagrams, logging evidence) that supports compliance documentation owned by the security team.
Serve as the organization's internal expert on applied AI tooling: stay current on the ecosystem, evaluate new capabilities, and translate them into concrete proposals.
Report periodically to leadership on friction points and opportunities — informing decisions rather than driving adoption targets.