Lead Software Platform Engineer, MLOps

TetraScience • San Mateo, CA

17h • $200,000 - $270,000 • Remote

About The Position

TetraScience is seeking a Lead Software Platform Engineer to work at the intersection of distributed systems and MLOps. The role owns and scales AI and data infrastructure as a multi-tenant product used by customers and internal engineering teams. The engineer will architect cloud-based services and MLOps infrastructure for production AI/ML workflows, partnering with Applied AI engineers, data engineers, and platform teams, and serving as the technical design authority for production models, LLMs, and agents. The work spans model and prompt lifecycle management, evaluation and observability, security and tenant boundaries, and cost and latency controls in a regulated environment.

The platform serves scientists at large pharmaceutical companies, who use it with their own data and compliance obligations. The role requires turning scalability and cost goals into technical strategy, ensuring production readiness, and guiding architecture and engineering practices across product and engineering.

Requirements

• 10+ years of professional experience in software engineering and infrastructure engineering, with a track record of designing, building, and scaling distributed cloud-native systems in production.
• Demonstrated technical leadership or architecture experience, with accountability for system design, scalability, performance, and cost optimization decisions.
• Experience designing security into a multi-tenant platform, including tenant authorization boundaries and sensitive data such as PII and PHI, with awareness of LLM-specific risks such as prompt injection and tool abuse.
• Extensive experience building and maintaining production AI/ML infrastructure and delivering model deployment and lifecycle management as a multi-tenant product for external users, rather than as internal tooling for one team.
• Deep hands-on experience taking LLM-based systems to production, including RAG architecture, retrieval and embedding design, prompt and model versioning, and tool or function calling, operated under real traffic, latency budgets, and failure conditions.
• Expert-level TypeScript and Python coding skills for robust APIs and backend services, and the judgment to evaluate AI-generated code for correctness, security, and architectural fit.
• Production-level experience with a model registry and serving stack, ideally Databricks MLflow, including model registration, versioning, asset bundles, and serving workflows.
• Experience treating AI evaluation as a release gate, including evaluation harnesses, regression gates, and drift or quality monitoring for non-deterministic systems.
• Proficiency in API-first design, including REST and OpenAPI specifications, and designing scalable, secure, versioned, and extensible APIs.
• Solid working knowledge of AWS and containerized workloads such as Docker, and familiarity with infrastructure-as-code such as CloudFormation or CDK; ability to contribute to CI/CD pipelines and deployment automation.
• Experience defining observability and SLI/SLO/SLA practices for production systems, including monitoring, alerting, and distributed tracing.
• Ability to communicate clearly with customers and cross-functional teams, influence technical direction on teams not directly managed, and mentor other engineers.

Nice To Haves

• Familiarity with emerging LLM frameworks for advanced prompt orchestration and programmatic LLM pipelines.
• Experience operating agentic or multi-step orchestration in production and using MCP as an integration surface for tooling and non-human identities.
• Understanding of LLM cost monitoring, latency optimization, and production usage analytics, including per-tenant attribution for tokens, GPU, and inference.
• Experience with multimodal model inputs, including image and instrument data, and the throughput and cost implications of serving them at scale.
• Experience with fine-tuning, distillation, or model optimization techniques such as quantization, batching, or KV-cache strategy to improve latency and cost.
• Experience delivering ML or AI systems in regulated or validated environments (GxP, 21 CFR Part 11, SOC 2), including computer system validation and audit readiness.
• Background in scientific, life sciences, or laboratory data domains.

Responsibilities

• Own the technical architecture of the AI/ML platform, including the customer-facing service and API surface and the interfaces used by Applied AI and data engineering teams.
• Own the end-to-end model and prompt lifecycle across Databricks MLflow and AWS Bedrock, including registration, versioning, asset bundles, staged promotion, rollback, and multi-model serving.
• Design the inference substrate for real-time and batch AI workloads, including routing, batching, caching, concurrency control, accelerator capacity planning, large binary inputs, and graceful degradation.
• Integrate AI models and LLMs into production systems using architectures such as RAG, and architect the agentic layer, including tool and function calling, MCP tooling, and agent runtimes.
• Design AI platform security with the security team, including guardrails, prompt-injection and tool-abuse defenses, PII and PHI handling, and tenant data boundaries across prompts, retrieval, and tool calls.
• Build evaluation and quality infrastructure, including offline and online evaluation harnesses, golden datasets, CI regression gates, A/B and shadow deployment, and production drift and hallucination detection.
• Establish AI platform observability, including monitoring, alerting, logging, distributed tracing, and SLI, SLO, and SLA practices for probabilistic systems.
• Design reproducibility and lineage for validated pharma environments, including versioned data, code, prompts, and model artifacts, and auditable records.
• Contribute to infrastructure-as-code and deployment automation using CloudFormation and AWS CDK, supporting multi-tenant infrastructure, online upgrades, and on-demand compute allocation with the production deployment team.
• Own production readiness with Applied AI engineers, data engineers, and platform teams, including performance, reliability, cost-efficiency, incident response, and runbooks.
• Act as subject-matter expert and design authority across product and engineering; lead design reviews, write reference architectures and technical documentation, and mentor senior and mid-level engineers.
• Set technical direction for emerging AI infrastructure by evaluating frameworks, serving runtimes, model providers, and data types, and making build-versus-buy decisions based on cost, risk, and scalability.

Benefits

• 100% employer-paid benefits for eligible employees and immediate family members.
• Unlimited paid time off (PTO).
• 401K.
• Flexible working arrangements, including remote work.
• Company-paid life insurance and LTD/STD.
• A culture of continuous improvement with career growth and coaching.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed