Staff Software Engineer, AI Inference

Syllo

18d•$190,000 - $230,000•Remote

About The Position

Syllo is hiring a Staff Software Engineer to lead development of its production AI inference platform. The role will define the architecture and technical roadmap for inference infrastructure, model serving, and runtime optimization, and build systems that deploy, optimize, monitor, and operate open-source language models in production. It is a hands-on technical position focused primarily on designing, building, and optimizing production systems while shaping Syllo’s long-term AI infrastructure strategy. The engineer will work closely with machine learning, infrastructure, and product engineering teams as the company scales its AI platform.

Requirements

Significant experience designing and operating production AI inference systems.
Experience building or leading production LLM serving infrastructure.
Deep experience with one or more modern inference runtimes and frameworks, such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, Hugging Face TGI, NVIDIA Dynamo, or comparable technologies.
Strong background in distributed systems, backend infrastructure, or high-performance platform engineering.
Experience optimizing inference performance across GPU workloads, including latency, throughput, batching, memory utilization, and serving efficiency.
Experience operating GPU infrastructure in production.
Strong proficiency in Python and at least one systems programming language, such as Go, Rust, or C++.
Proven ability to lead technical architecture for complex infrastructure initiatives.
Excellent communication skills and ability to influence technical direction across engineering teams.

Responsibilities

Lead the design and development of the production inference platform.
Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
Build and operate scalable, cost-effective systems for serving large language models in production.
Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
Partner with ML engineers to productionize new models and inference techniques.
Establish benchmarking methodologies for evaluating models, runtimes, and hardware.
Make architectural decisions about whether to build internally or use open-source or commercial solutions.
Mentor engineers as the team grows and establish engineering best practices for AI infrastructure.