AI Researcher (Remote)

Cisco•San Francisco, CA
•$216,300 - $280,800•Hybrid

About The Position

The Splunk Agent Resilience team develops scalable, cost-effective evaluation and guardrails to improve the reliability of AI agents and reduce risks, helping customers deploy and manage AI-powered applications with greater observability and control. The AI Researcher independently owns applied research problems focused on understanding, evaluating, and improving AI agents across complete workflows. The work spans agent evaluation, behavioral failure discovery, agent understanding, simulation, and optimization. The researcher develops algorithms and research methods, validates them through rigorous experimentation, and builds generalizable AI solutions that balance quality, latency, and cost. The role involves constructing datasets and benchmarks, incorporating human judgment, and validating ideas through controlled experiments, strong baselines, ablations, and systematic error analysis. The researcher works with researchers, Product, Engineering, and customers to take research from hypothesis to reproducible results and useful product capabilities. The position is hybrid in the Bay Area, New York City, or Seattle, Washington; fully remote candidates within the United States will also be considered. Occasional meetings outside standard local business hours may be needed to accommodate global teams and customers.

Requirements

  • Bachelor's degree in Computer Science, Machine Learning/Artificial Intelligence, Mathematics, or a related field and 8+ years of related experience; or a master's degree and 6+ years of related experience; or a PhD and 3+ years of related experience.
  • Experience applying machine learning or AI research to deployed products or practical business problems, including independently leading research workstreams from problem definition through experimentation, evaluation, and implementation.
  • Experience training and fine-tuning encoder or decoder transformer models using supervised fine-tuning or parameter-efficient methods such as LoRA.
  • Experience developing machine learning solutions with Python and deep learning frameworks such as PyTorch, TensorFlow, or equivalent, including model training, optimization, and evaluation.
  • Experience developing and evaluating machine learning datasets, including data sourcing, curation, labeling, synthetic or simulation-based data generation, and experimental validation.

Nice To Haves

  • Master's degree or PhD in Machine Learning, Computer Science, Statistics, or a related field.
  • Experience designing and executing controlled experiments, including baseline comparisons, ablation studies, leakage prevention, statistical analysis, and systematic error analysis.
  • Contributions to novel modeling, training, evaluation, or data generation methods demonstrated through research publications, open-source implementations, or documented experimental results.
  • Experience with advanced model training techniques, including preference optimization, contrastive learning, multi-task learning, or approaches beyond supervised fine-tuning.
  • Experience developing simulation and synthetic data pipelines, optimizing data diversity and difficulty, or improving annotation consistency and label quality.
  • Experience with distributed model training, efficient processing of large datasets, or optimization of model training and inference performance.

Responsibilities

  • Independently lead applied research workstreams from hypothesis development and dataset preparation through model training, evaluation, and implementation.
  • Train and fine-tune small language models using supervised fine-tuning, LoRA, and other post-training techniques.
  • Develop and curate large-scale training datasets using real-world data, simulation, and synthetic data generation; establish labeling guidelines, analyze annotation disagreements, and validate data quality.
  • Research and implement improvements to training objectives, data mixtures, sampling strategies, and model adaptation methods, applying research literature and developing new methods.
  • Design and execute controlled experiments, baseline comparisons, and ablation studies to evaluate model performance, robustness, generalization, and failure modes.
  • Develop reproducible model training and evaluation pipelines and document methodologies and findings for continued research and product development.
  • Collaborate with Product Management and Engineering to evaluate model behavior, address deployment constraints, and integrate validated research methods into production workflows.
  • Communicate technical findings through research code, experimental reports, model documentation, and discussions with technical and non-technical stakeholders.
  • Collaborate with global teams and customers, including occasional meetings outside standard local business hours to accommodate different time zones.

Benefits

  • Medical insurance
  • Dental insurance
  • Vision insurance
  • 401(k) plan with Cisco matching contribution
  • Paid parental leave
  • Short- and long-term disability coverage
  • Basic life insurance
  • Paid holidays
  • Floating holiday for non-exempt employees
  • Paid birthday day off
  • Paid year-end holiday shutdown
  • Paid vacation or flexible vacation time off, depending on employee classification
  • Paid sick time
  • Additional paid time away may be requested for critical or emergency family issues
  • Optional paid volunteer days
  • Annual bonuses for non-sales roles, subject to Cisco policies
  • Potential Cisco restricted stock unit grants, subject to eligibility and plan terms
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service