Senior AIOps Engineer

Porch Group•Seattle, WA

5d•$135,000 - $189,000•Remote

About The Position

Porch Group is hiring a remote Senior AIOps Engineer I to help ensure its AI- and ML-powered systems are reliable, observable, secure, and cost-efficient in production. The engineer will collaborate with product managers, platform engineers, data scientists, and machine learning engineers to operate and evolve AI infrastructure and help teams deliver AI features safely and efficiently. The AI/ML stack is built with Python and runs on Kubernetes (GKE) and Google Cloud Platform. It uses tools including Union Cloud (Flyte), BentoML, Feast, Label Studio, BigQuery, and Dataflow. GenAI systems include a centralized LLM routing and gateway service, batch prediction services for large-scale inference, and developing RAG infrastructure. The role covers production reliability, monitoring and model lifecycle operations, deployment automation, AI infrastructure, data annotation platform operations, RAG and vector database infrastructure, and LLM cost management. It includes participation in an on-call rotation and incident response. The position is remote in the United States, except for specified states and jurisdictions from which Porch Group does not consider remote applicants. The listed annual pay range is $135,000 to $189,000, with eligibility for long-term incentive awards subject to program guidelines and approvals.

Requirements

At least 4 years of applicable experience, including at least 3 years in commercial software engineering or production operations such as SRE, DevOps, Platform, or ML Platform, operating services on Kubernetes and/or major cloud platforms.
Experience operating production systems at scale, preferably distributed, microservices-based, or Kubernetes-based architectures.
Hands-on experience with CI/CD tools and deployment automation such as GitLab CI/CD, GitHub Actions, or similar.
Hands-on experience with infrastructure-as-code tools, preferably Terraform; monitoring, logging, and alerting tools; and container management and the Docker image lifecycle.
Experience operating and administering self-hosted open-source tools on Kubernetes, including installation, configuration, upgrades, security hardening, backup, and disaster recovery.
Understanding of basic machine learning concepts and model lifecycles, including training, evaluation, deployment, and monitoring; familiarity with common ML tooling.
Strong computer science and programming fundamentals, problem-solving skills, and ability to write high-quality software using TDD, SOLID principles, clean code, and robust documentation.
Ability to communicate effectively with data scientists, ML engineers, and product stakeholders in business and technical contexts.
Comfort participating in an on-call rotation and working in a trustful, respectful, and collaborative environment.
Curious and driven mindset, with a strong interest in how AI systems behave in production and how to improve their reliability and observability.

Nice To Haves

Strong GCP experience is preferred, including hands-on work with GKE, BigQuery (data warehousing, SQL, and schema management), Pub/Sub, Vertex AI (batch prediction and model deployment), Cloud SQL, GCS, and IAM.
Familiarity with LLM ecosystem tooling such as AI gateways, vector databases, RAG pipelines, embedding models, or LLM evaluation frameworks.
Experience with Python-based data and ML stacks such as NumPy, SciPy, Pandas, Scikit-Learn, and PyTorch.
Experience building or operating ML infrastructure such as feature stores, orchestration platforms, model serving, and annotation tools is highly desirable.

Responsibilities

Own production reliability for AI/ML services by monitoring availability and performance, defining user- and business-impact-focused SLOs and SLIs, and managing recurring model refresh cycles, including retraining, validation, and redeployment.
Design and implement monitoring, logging, and alerting for models and data pipelines, and integrate model and system metrics with observability stacks and dashboards such as Datadog and Opik.
Streamline AI-service deployment workflows with data scientists and ML engineers, including blue/green, canary, A/B, and shadow deployments; improve CI/CD and release processes; and support productionization of image-based ML models and batch prediction workflows.
Provision, deploy, configure, and maintain adopted AI infrastructure, including AI gateways, RAG platforms, LLM observability tools, agentic workflows, and no-code agent builders.
Build automation and tooling to reduce manual work in model promotion, configuration, environment management, and Docker image maintenance; support multi-business-unit infrastructure provisioning and CI/CD integrations.
Own the operational health of Label Studio, including supporting GCS buckets, BigQuery pipelines, database and SSO dependencies, and bi-weekly annotation project cycles.
Stand up and operate RAG and vector database infrastructure, including embedding pipelines and context retrieval APIs, and optimize embedding storage and retrieval performance at scale.
Monitor and optimize LLM token usage and costs across providers; maintain centralized cost-tracking dashboards and alerts across business units; and recommend cost-efficient model routing and provider strategies.
Participate in an AI/ML on-call rotation, triage and resolve production incidents, and lead or contribute to post-incident reviews that drive lasting improvements.
Develop production-grade services and tooling, primarily in Python on Kubernetes, using strong engineering practices such as testing, code reviews, CI/CD, observability, and security.
Document standards, architecture diagrams, and operational playbooks for other teams.

Benefits

Eligibility for long-term incentive awards, subject to program guidelines and approvals.
Three medical plan options, two dental plan options, and a vision plan.
Voluntary Critical Illness, Hospital Indemnity, and Accident plans.
Pre-tax savings options, including a partially employer-funded Health Savings Account and employee Flexible Spending Accounts for healthcare, dependent care, and transportation.
Company-paid Basic Life and AD&D, and Short- and Long-Term Disability benefits; voluntary Life and AD&D plans are also offered.
Traditional and Roth 401(k) plans with a discretionary employer match.
Employer-paid wellbeing program offering guided meditation and mindfulness exercises, mental health coaching, clinical care, and confidential resources including will preparation.
Free LifeBalance resource with year-round discounts, including gym memberships, travel, appliances, movies, and pet insurance.
Flexible paid vacation, typically nine company-paid holidays per year, paid sick time, and paid parental leave.
Identity theft program, travel assistance, and fitness and other discount programs.