AI Automation Engineer

Fortinet•Sunnyvale, CA

About The Position

Fortinet is seeking an AI Automation Engineer to build and maintain its technical interview pipeline, including the question bank, AI-assisted scoring, and workflow automation. The role involves designing systems for adding and calibrating interview questions and evaluating candidate answers, with an emphasis on consistent AI prompting, validating output quality against ground truth, and identifying and correcting system errors. The position is expected to expand over time into broader automation work, though the scope of that growth is still being defined. The engineer will manage the assessment repository, develop and benchmark scoring workflows, automate the interview process from question selection through reporting, and monitor quality and fairness. The role also includes documenting prompts and scoring logic for auditability, helping move source code and repositories to an internal Git platform, and partnering with engineering teams to identify other internal automation opportunities. Fortinet offers medical, dental, vision, life and disability insurance, a 401(k), 11 paid holidays, vacation and sick time, and a comprehensive leave program. The expected earnings range is $110,000–$170,000. Roles are eligible for the Fortinet equity program; bonus eligibility is reviewed at hire and annually at the Company’s discretion.

Requirements

Bachelor's degree in Computer Science or a related field, or equivalent practical experience.
Solid programming fundamentals in Python, JavaScript/TypeScript, or a similar language.
Comfort working with APIs, including LLM/AI APIs such as OpenAI or Anthropic.
Demonstrated experience designing and evaluating prompts or pipelines for LLM-based tools, beyond simply using AI products.
Proficiency with Git and a hosted platform such as GitHub or GitLab, including branching, PRs/MRs, code review, and CI basics.
Basic understanding of databases such as SQL.
Strong attention to detail and technical writing ability.
Ability to work independently on processes that are still being defined and adapt as scope evolves.

Nice To Haves

Familiarity with test or assessment platforms or ATS integrations.
Exposure to scripting automation, such as workflow tools, cron jobs, or CI-style pipelines.
Interest or coursework in fairness or bias in ML systems.
Prior experience conducting or coordinating technical interviews.

Responsibilities

Add, revise, and retire technical interview questions across roles, levels, and skill areas.
Maintain question metadata, including difficulty, topic tags, expected answer criteria, and role relevance.
Create question variants or randomized versions to reduce answer-sharing risk.
Build automated and manual validation checks to identify incorrect, ambiguous, or miscalibrated questions before repository entry.
Automate updates to problem sets and recalculate average difficulty ratings as results accumulate.
Periodically review questions to prevent leakage or staleness and keep content aligned with job requirements.
Architect scoring workflows that compare candidate answers with reference answers and rubrics using a clear methodology.
Calibrate and benchmark scoring prompts and models against human-graded samples, measuring accuracy and identifying systematic errors or bias.
Design safeguards for model failure modes such as inconsistency, hallucination, and prompt sensitivity.
Flag edge cases or low-confidence scores for human review instead of fully automating high-stakes decisions.
Automate the workflow from question selection through test delivery, answer collection, scoring, and reporting.
Help transition source code and repositories to an internal Git platform.
Track test results to help establish scoring benchmarks over time.
Monitor scoring consistency, accuracy, and pipeline health.
Watch for and mitigate AI evaluation bias across candidate demographics or answer styles.
Document processes, prompts, and scoring logic for auditability.
Apply AI integration and automation skills to additional internal workflows as the role expands.
Partner with engineering teams to identify manual, repetitive processes suitable for automation.
Take on additional internal tooling and infrastructure projects as capacity and scope grow.

Benefits

Medical insurance
Dental insurance
Vision insurance
Life insurance
Disability insurance
401(k)
11 paid holidays
Vacation time
Sick time
Comprehensive leave program
Fortinet equity program eligibility
Potential bonus eligibility, reviewed at hire and annually at the Company's discretion