Ml Engineer Jobs

1,484 jobs found — updated daily

Staff II Software Engineer AI/ML Ops

BlackLine•Pleasanton, CA
•$245,000 - $307,000•Hybrid

About The Position

The Staff II Software Engineer will lead the design, deployment, integration, and operation of AI, machine learning, and LLM-based systems. The role works across data science, security, product, and development teams to establish governance standards, build reliable ML and agentic infrastructure, and move models and AI solutions from research into production. The position includes architecting model registries, feature stores, agent runtimes, evaluation frameworks, observability, and end-to-end guardrails. It also involves managing cloud-based training and inference infrastructure, automating CI/CD and ML workflows, monitoring model and agent performance, and ensuring security, auditability, privacy, and regulatory compliance. The engineer will provide technical leadership, mentor engineers, drive design reviews, and lead incident response and reliability strategies for AI/ML systems. BlackLine describes its workplace as a combination of virtual and in-person interactions. Candidates who live within a reasonable commute of an office are expected to work there at least three days per week.

Requirements

  • Extensive practical experience with demonstrated success developing effective business solutions or applications for products or services that may affect broad areas of the organization.
  • Recognized expertise and industry knowledge as an individual contributor to operations.
  • Ability to set objectives and deliver results affecting a department or division.
  • Ability to provide advice, counsel, and thought leadership within a department.
  • High-level strategic influence, with decisions that affect a business unit's or department's strategic direction.
  • Ability to anticipate emerging trends and take a forward-looking, multi-year perspective.
  • High degree of autonomy and independent discretion when handling complex, strategic duties requiring functional expertise.
  • Ability to navigate ambiguous organizational challenges and develop innovations or adaptations.

Nice To Haves

  • Bachelor's or master's degree in Computer Science, Machine Learning, Data Science, or a related field.
  • Strong programming skills in Python, Java, or Scala.
  • Expertise with ML frameworks such as TensorFlow, PyTorch, or scikit-learn and orchestration tools such as Airflow, Kubeflow, Vertex AI, or MLflow.
  • Experience operating production pipelines for ML and LLM-based systems across GCP, AWS, or Azure.
  • Familiarity with LangChain, LangGraph, ADK, or similar agentic system runtime management.
  • Strong competencies in CI/CD, Infrastructure as Code, and DevSecOps pipelines integrating testing, compliance, and deployment automation.
  • Hands-on experience with observability stacks such as Prometheus, Grafana, or New Relic for model and agent performance tracking.
  • Understanding of Responsible AI governance, auditability, and cost metering for training and inference workloads.
  • Proficiency with containerization technologies such as Docker and Kubernetes.
  • Proficiency in scripting languages such as Bash or Python for automation.
  • Experience with workflow orchestration tools such as Apache Airflow.
  • Expertise managing and optimizing cloud-based infrastructure.
  • Familiarity with DevOps practices and tools for automated deployment.
  • Understanding of network configurations and security protocols.
  • Ability to define problems, collect and analyze data, and propose innovative solutions.
  • Strong critical-thinking skills for evaluating models and identifying limitations.
  • Adaptability in a fast-paced, rapidly evolving environment and proactive learning about trends, techniques, and technologies in AI and data science.

Responsibilities

  • Partner with data science, security, and product teams to establish evaluation and governance standards for guardrails, bias, drift, and latency SLAs.
  • Mentor senior engineers and lead design reviews for ML pipelines, model registries, and agentic runtime environments.
  • Lead incident response and reliability strategies for ML and AI systems.
  • Collaborate with development teams to integrate AI solutions into existing workflows, applications, platforms, and technologies.
  • Define and manage an MCP Registry for agentic component onboarding, lifecycle versioning, and dependency governance.
  • Build CI/CD pipelines for LLM agent deployment, policy validation, and prompt evaluation.
  • Develop experimentation frameworks for agent evaluations, scenario regression, and performance analytics.
  • Implement logging, metering, and auditing for agent behavior and function calls, and support compliance alignment.
  • Create observability systems tracking conversation outcomes, factual accuracy, latency, escalation patterns, and safety events.
  • Architect AI-agent guardrails, including prompt-injection protection, identity-aware routing, and tool-use authorization.
  • Collaborate across teams to standardize authentication, authorization, and session governance for multi-agent runtimes.
  • Architect and standardize model registries and feature stores for version tracking, lineage, and reproducibility.
  • Lead deployment of machine learning models into production environments, ensuring scalability, reliability, and efficiency.
  • Collaborate with software engineers to integrate models into applications and systems, and implement and maintain model-inference APIs.
  • Design and manage training infrastructure, including distributed training orchestration, GPU/TPU allocation, and automatic scaling.
  • Implement CI/CD for model workflows, including model validation, bias checks, and rollback automation.
  • Build reproducible experimentation frameworks for training, tuning, and deployment cycles.
  • Manage and optimize cloud infrastructure required for machine learning operations, supporting system availability, security, and performance.
  • Monitor deployed models and work with data scientists and engineers to resolve model-performance and data-quality issues.
  • Perform maintenance, updates, and optimizations for machine learning solutions.
  • Develop automation scripts and tools and implement orchestration systems for the ML lifecycle, from data preparation through model deployment.
  • Work with data scientists to understand deployment requirements and constraints and transition models from research to production.
  • Analyze production performance and resource utilization and implement improvements to model efficiency.
  • Implement security measures and support regulatory and industry-standard compliance for ML systems and data privacy.
  • Integrate audit controls, metadata storage, and lineage tracking across ML and AI workflows.
  • Establish monitoring and feedback loops, including event logs, evaluations, and automated retraining triggers.
  • Enforce secure deployment patterns using Infrastructure as Code and cloud-native secrets management.
  • Define SLAs, error budgets, and compliance reporting mechanisms for ML and AI systems.

Benefits

  • Professional development seminars.
  • Inclusive affinity groups.
  • Short-term and long-term incentive programs, based on eligibility.
  • Benefit and wellness plans.

Build a Resume for Ml Engineer

The resume builder that gets results.

  • Get clear feedback so you look as qualified as you are
  • Align your resume with the job to get further in the process, faster
  • Take the guesswork out of resume writing

Explore Related Job Searches

© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service