Sr. Backend Software Engineer (Infrastructure ML Ops)

Philo•San Francisco, CA

About The Position

Philo is seeking a senior backend software engineer to build and improve the infrastructure supporting machine learning and large language model workloads. Philo’s infrastructure team provides the foundation for a 24/7 live and on-demand streaming product and a platform used by teams across the company. Its distributed, horizontally scalable systems run in the cloud on Kubernetes, with continuous testing and deployment and close monitoring of production health and performance. The role will help set direction and establish best practices for Philo’s ML and LLM infrastructure. The engineer will build serving, evaluation, and platform systems for workloads such as content recommendations, internal coding and information agents, and video-ad classification and catalog labeling. The infrastructure supports real-time recommendation models and LLM workloads, with an emphasis on reliability, scalability, security, observability, and cost-effective operation. The position works closely with Philo engineers and internal stakeholders, as well as third-party vendors, to develop and operate cloud infrastructure, model-serving and evaluation systems, CI/CD capabilities, and tools for teams using the platform. The role is language agnostic; much of Philo’s backend code is written in Go, Ruby, and TypeScript, with some C++ and Python. Services run on Kubernetes, and Philo practices continuous deployment.

Requirements

At least 8 years of software development and infrastructure management experience.
At least 4 years of experience developing and deploying ML systems at scale across training and inference.
Experience with distributed systems and an understanding of microservices architecture principles.
Experience with Linux and containerized environments such as Docker.
Experience managing cloud computing environments (AWS or GCP) and configuring cloud services such as CloudWatch, Route 53, RDS, ElastiCache, SQS, load balancers, VPC networking, and IAM security.
Experience with container orchestration platforms such as Kubernetes and familiarity with running GPU or accelerated workloads on them.
Strong understanding of networking and internet application protocols, including TCP/IP, DNS, and HTTP.
Strong understanding of network and application security principles and best practices.
Proficiency in writing, testing, and profiling software in Go, JavaScript/TypeScript, C++, Ruby, Python, or similar languages.
Experience and aptitude for collaborating and communicating with internal and external stakeholders in business and technical roles.

Nice To Haves

Hands-on experience optimizing inference for low-latency or high-throughput serving, such as dynamic or continuous batching, quantization, KV-cache management, or GPU memory tuning.
Experience with ML platform tooling, including model registries and experiment tracking, feature stores, and pipeline or workflow orchestration.
Experience building model evaluation, monitoring, or experimentation systems, including drift detection, LLM-as-judge, or model A/B testing.
Experience with LLMOps, including inference gateways or routers, hosted model APIs, RAG pipelines, vector databases, and prompt or version management.
Experience building or operating agent harnesses or tool-using LLM applications in production.
Experience with data and model versioning and data or labeling pipelines.
Experience with cost management or FinOps for GPU and inference workloads.
Familiarity or hands-on experience with configuration management systems and Infrastructure as Code, such as Terraform or CloudFormation.
Familiarity or hands-on experience with monitoring and observability systems such as Prometheus, Grafana, TICK/InfluxDB, Fluentd, ELK, or Datadog.
Familiarity or hands-on experience with CI/CD automation systems such as Jenkins or GitLab.
Experience with relational and non-relational databases and familiarity with modern data warehousing and querying.

Responsibilities

Manage Philo’s core infrastructure with a focus on supporting ML and LLM workloads used by subscribers and internal stakeholders.
Design, build, and maintain cloud infrastructure components on AWS and Kubernetes using in-house technology and open-source software.
Build and scale model-serving and inference infrastructure for real-time recommendation models and LLM workloads, including model servers, inference engines, autoscaling, and latency and throughput optimization.
Manage GPU and accelerator infrastructure on Kubernetes, including scheduling, capacity and quota management, and cost optimization across on-demand and spot capacity.
Design, build, and maintain monitoring and observability systems for system health and model quality, including drift, evaluation, and online performance.
Build and operate model-evaluation infrastructure, including offline and online evaluation pipelines, golden and regression datasets, LLM-as-judge harnesses, A/B testing, experimentation, and feedback and ground-truth loops.
Design, build, and maintain CI/CD systems that enable engineers to create pipelines to test and deploy code.
Establish tools, methods, and best practices for engineers using Philo’s ML infrastructure.
Ensure the reliability, security, and scalability of the Philo platform.
Promote Infrastructure as Code.
Work with other Philo engineers to deploy and instrument software systems.
Drive evaluation, selection, and integration of third-party vendor systems, and work with vendors to configure and manage them.

Benefits

Full health coverage for employees and their families.
Dental insurance.
Vision insurance.
401(k) plan with employer contributions.
Flexible working hours.
Up to 20 weeks of fully paid parental leave.
Unlimited paid time off for vacation and sick leave.
Annual vacation bonus of $2,000 for taking a two-week vacation.
$5,250 annually for professional development and educational assistance.
Home office and TV stipend: $1,250 in the first year and $250 annually thereafter.
Monthly bonus for employees who commit to working at least three days per week in an office.
Commuter benefits for transit, rideshare, bike rental, or parking at the San Francisco headquarters.
Free Gympass subscription, offering access to gyms, studios, classes, training, and wellness apps.
Dog-friendly office.
Company stock options.