Sr. Backend Software Engineer (Infrastructure ML Ops)

Philo•Cambridge, MA

21h•$165,000 - $250,000

About The Position

Philo is seeking an Infrastructure MLOps Software Engineer to build and improve the machine-learning infrastructure supporting its streaming product and internal teams. The infrastructure team provides a cloud-based, highly distributed and horizontally scalable foundation running on Kubernetes. The role will help set direction, adopt best practices, and deliver infrastructure used for ML and LLM workloads across the company, including content recommendations, internal coding and knowledge agents, and video classification and labeling. The engineer will build and operate model serving, inference, evaluation, and platform infrastructure. This includes cloud infrastructure on AWS and Kubernetes, GPU and accelerator capacity, model-quality and system observability, evaluation and experimentation pipelines, and CI/CD systems. The role also establishes tools and practices for engineers using the platform and works with internal teams and third-party vendors to deploy, instrument, and operate reliable, secure, scalable systems. Philo is headquartered in San Francisco, with offices in New York City and Cambridge, Massachusetts. This role may be based in San Francisco, Brooklyn, Cambridge, or remotely. It is full-time. Compensation includes annual salary, company stock options, and health benefits; salary varies by experience and location.

Requirements

At least 8 years of software development and infrastructure management experience.
At least 4 years of experience developing and deploying ML systems at scale across training and inference.
Experience with distributed systems and understanding of microservices architecture principles.
Experience with Linux and containerized environments such as Docker.
Experience managing AWS or GCP cloud environments and configuring cloud services such as CloudWatch, Route 53, RDS, ElastiCache, SQS, ALB/NLB/ELB, VPC networking, and IAM security.
Experience with Kubernetes or other container orchestration platforms, and familiarity with GPU or accelerated workloads on them.
Strong understanding of networking and internet application protocols, including TCP/IP, DNS, and HTTP.
Strong understanding of network and application security principles and best practices.
Familiarity or hands-on experience with configuration management and Infrastructure as Code systems such as Terraform or CloudFormation.
Familiarity or hands-on experience with monitoring and observability systems such as Prometheus, Grafana, TICK/InfluxDB, Fluentd, ELK, or Datadog.
Familiarity or hands-on experience with CI/CD automation systems such as Jenkins or GitLab.
Experience with relational and non-relational databases and familiarity with modern data warehousing and querying.
Proficiency in writing, testing, and profiling software in Golang, JavaScript/TypeScript, C++, Ruby, Python, or similar languages.
Ability and aptitude to collaborate and communicate with internal and external stakeholders in business and technical roles.

Nice To Haves

Hands-on experience optimizing inference for low latency or high throughput, including dynamic or continuous batching, quantization, KV-cache management, or GPU memory tuning.
Experience with ML platform tooling such as model registries and experiment tracking, feature stores, and pipeline or workflow orchestration.
Experience building model evaluation, monitoring, or experimentation systems, including drift detection, LLM-as-judge, or A/B testing for models.
Experience with LLMOps, including inference gateways or routers, hosted model APIs, RAG pipelines, vector databases, and prompt or version management.
Experience building or operating agent harnesses or tool-using LLM applications in production.
Experience with data and model versioning and data or labeling pipelines.
Experience with cost management or FinOps for GPU and inference workloads.

Responsibilities

Manage Philo’s core infrastructure, with a focus on supporting ML and LLM workloads used by subscribers and internal stakeholders.
Design, build, and maintain cloud infrastructure components using AWS, Kubernetes, in-house technology, and open-source software.
Build and scale model-serving and inference infrastructure for real-time recommendation models and LLM workloads, including model servers, inference engines, autoscaling, and latency and throughput optimization.
Manage GPU and accelerator infrastructure on Kubernetes, including scheduling, capacity and quota management, and cost optimization across on-demand and spot capacity.
Design, build, and maintain monitoring and observability systems for system health and model quality, including drift, evaluation, and online performance.
Build and operate model-evaluation infrastructure, including offline and online evaluation pipelines, golden and regression datasets, LLM-as-judge harnesses, A/B testing, experimentation, and feedback and ground-truth loops.
Design, build, and maintain CI/CD systems that enable engineers to test and deploy code.
Establish tools, methods, and best practices for engineers working with Philo’s ML infrastructure.
Ensure platform reliability, security, and scalability, and promote Infrastructure as Code.
Work with other Philo engineers to deploy and instrument software systems.
Evaluate, select, and integrate third-party vendor systems, and work with vendors to configure and manage them.

Benefits

Full health, dental, and vision coverage for employees and their families.
401(k) plan with employer contributions: 100% match on deferrals up to 3% of pay and 50% match on the next 2%.
Flexible working hours.
Up to 20 weeks of fully paid parental leave.
Unlimited paid time off for vacation and sick leave.
$2,000 annual vacation bonus for taking a two-week vacation.
$5,250 annually for professional development and educational assistance.
$1,250 annual home office and TV stipend during the first year, and $250 annually thereafter.
$500 per month bonus for employees who commit to working at least three days per week in an office.
Commuter benefits of $315 per month toward transit, rideshare, bike rental, or parking at the San Francisco headquarters.
Free Gympass subscription.
Dog-friendly office.
Company stock options.