AI Automation Engineer

Fortinet•Sunnyvale, CA

12h•$110,000 - $170,000

About The Position

Fortinet is seeking a highly motivated AI Automation Engineer to build and maintain its technical interview pipeline, including the question bank, AI-assisted scoring, and workflow automation. The role involves designing systems for adding and calibrating interview questions and evaluating candidate answers. The engineer will structure and prompt AI models for consistent results, measure and validate output quality against ground truth, and identify and correct system errors. The role is expected to expand into broader automation work, though its future scope is still being defined. The position also applies AI integration and automation skills to other internal workflows as the role grows, partnering with engineering teams to identify repetitive processes for automation and taking on additional tooling and infrastructure projects as capacity and scope allow.

Requirements

Bachelor's degree in Computer Science or a related field, or equivalent practical experience.
Solid programming fundamentals in Python, JavaScript/TypeScript, or a similar language.
Comfort working with APIs, including LLM/AI APIs such as OpenAI or Anthropic.
Demonstrated experience designing and evaluating prompts or pipelines for LLM-based tools, beyond simply using AI products.
Proficiency with Git and a hosted platform such as GitHub or GitLab, including branching, PRs/MRs, code review, and CI basics.
Basic understanding of databases such as SQL.
Strong attention to detail and technical writing ability.
Ability to work independently on processes that are still being defined and adapt as scope evolves.

Nice To Haves

Familiarity with test or assessment platforms or ATS integrations.
Exposure to scripting automation, such as workflow tools, cron jobs, or CI-style pipelines.
Interest or coursework in fairness or bias in ML systems.
Prior experience conducting or coordinating technical interviews.

Responsibilities

Add, revise, and retire technical interview questions across roles, levels, and skill areas.
Maintain question metadata such as difficulty, topic tags, expected answer criteria, and role relevance.
Create question variants or randomized versions to reduce answer-sharing risk.
Build automated and manual validation checks to identify incorrect, ambiguous, or miscalibrated questions before they enter the repository.
Automate updates to problem sets and recalculate average difficulty ratings as results come in.
Periodically review and update questions to prevent leakage or staleness and keep content aligned with job requirements.
Design scoring workflows to compare candidate answers with reference answers and rubrics using a clear evaluation methodology.
Calibrate and benchmark scoring prompts and models against human-graded samples, measuring accuracy and identifying systematic errors or bias.
Design safeguards for model failure modes such as inconsistency, hallucination, and prompt sensitivity.
Flag edge cases and low-confidence scores for human review rather than fully automating high-stakes decisions.
Automate the workflow from question selection through test delivery, answer collection, scoring, and reporting.
Help transition source code and repositories to an internal Git platform.
Track test results to establish scoring benchmarks over time.
Monitor scoring consistency and accuracy, and report on pipeline health.
Monitor and mitigate AI evaluation bias across candidate demographics or answer styles.
Document processes, prompts, and scoring logic for auditability.
Apply automation and AI-integration skills to other internal workflows as the role expands.
Partner with engineering teams to identify manual, repetitive processes suitable for automation.
Take on additional internal tooling and infrastructure projects as capacity and scope grow.