Principal Machine Learning Engineer - CX

Cisco•San Jose, CA
•$296,300 - $374,800•Hybrid

About The Position

Cisco’s CX AI Incubation and Agents team owns the foundational AI platform behind intelligent customer experiences across Cisco. The shared platform supports application and agent teams in cloud, on-premises, and fully air-gapped customer environments. Its work includes AI-native platforms, model and inference layers, evaluation systems for shipped AI behavior, agentic and orchestration infrastructure, foundational AI services and APIs, and packaging and upgrade capabilities, as well as solutions for delivery intelligence, network automation, and edge computing. The team works across classical machine learning, domain-specific language models, and Agentic AI. As a Principal Machine Learning Engineer, you will lead ambiguous, high-impact cross-functional work from architecture and design through production and customer deployment. You will serve as an architectural and technical authority for the AI foundation powering Cisco IQ and evolving customer needs. This hands-on technical leadership role designs and builds production-scale AI systems and sets standards for system design, platform governance, and technical excellence across the CX engineering ecosystem. The role is hybrid and requires three days per week on site in San Jose, California. The application window is expected to close on December 17, 2026.

Requirements

  • Bachelor’s degree and 14 years of related experience, or master’s degree and 11 years of related experience, or PhD and 7 years of related experience.
  • Experience designing and building production-grade, multi-tenant distributed systems in Go, Python, or similar, including high-throughput service APIs, asynchronous/event-driven architectures, observability, and reliability engineering at scale.
  • Experience architecting and deploying production LLM and/or SLM inference runtimes optimized with continuous batching, KV-cache management, or speculative decoding.
  • Experience architecting and deploying production RAG systems and agentic workflows using vector databases.
  • Experience designing and implementing automated AI system evaluation frameworks, including golden test datasets, LLM-as-a-judge methodologies, drift detection, and/or CI/CD regression gates.

Nice To Haves

  • Core contributions to open-source inference, serving, or agent-orchestration projects such as vLLM, SGLang, Triton, or LangGraph.
  • Hands-on experience optimizing inference workloads for resource-constrained compute environments, including CPU, NPU, small GPU footprints, and on-premises hardware appliances.

Responsibilities

  • Design and build robust, multi-tenant AI services with scalable APIs, versioning, backward compatibility, and quota management across cloud, on-premises, and air-gapped environments.
  • Select, evaluate, and integrate inference runtimes, orchestration frameworks, and serving technologies.
  • Translate emerging AI capabilities into production architectures optimized for scalability, latency, memory, and compute across GPU, CPU, and edge environments.
  • Design and implement AI workload cost optimization through token efficiency, model selection, inference performance, and compute utilization; establish measurable engineering standards for cost efficiency.
  • Lead architecture and design reviews and resolve complex technical trade-offs.
  • Mentor senior engineers and influence platform direction through hands-on technical leadership and execution.
  • Lead cross-functional engagements from architecture and design through production and customer deployment.

Benefits

  • Medical insurance
  • Dental insurance
  • Vision insurance
  • 401(k) plan with Cisco matching contribution
  • Paid parental leave
  • Short- and long-term disability coverage
  • Basic life insurance
  • Paid holidays
  • Floating holiday for non-exempt employees
  • Paid birthday day off
  • Paid year-end holiday shutdown
  • Paid personal wellness days
  • Paid vacation or flexible vacation time off, depending on employee classification
  • Paid sick time
  • Additional paid time away may be requested for critical or emergency family issues
  • Optional paid volunteer days
  • Annual bonuses for eligible non-sales employees
  • Potential Cisco restricted stock unit grants
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service