Staff Engineer, Applied AI & Build Standards

Best Buy • Richfield, MN

3d • Hybrid

About The Position

As a Staff Engineer on Best Buy’s Applied AI & Build Standards team, internally known as AI Rails, you’ll develop evaluation frameworks, quality standards, and release controls to help ensure AI agents perform safely and effectively. The team builds a shared platform and sets build standards so Best Buy engineers and technologists can use AI agents with appropriate access, security, and cost controls. You’ll work across the platform, helping turn successful agent interactions into reusable capabilities and trusted organizational knowledge that supports future innovation.

This is a hybrid role. The employee is required to work on site Monday through Thursday at Best Buy’s corporate office in Richfield, Minnesota, and may work virtually from home or another non-Best Buy location on Friday. The recruiter or hiring manager can provide additional details during the hiring process. The role is eligible for immigration sponsorship.

Requirements

• At least 5 years of relevant experience with a bachelor’s degree, or equivalent relevant professional experience.
• At least 3 years of professional software development experience in Python.
• At least 2 years of experience building production data pipelines or search and retrieval systems, including data quality checks.
• At least 1 year of experience with formal or informal leadership.
• Hands-on experience building evaluation frameworks for LLM-based applications or AI agents.
• Daily, hands-on use of AI coding agents to write, test, and review code.

Nice To Haves

• Bachelor’s degree in IT, Computer Science, Engineering, or a related field.
• At least 7 years of relevant professional experience.
• Experience running evaluation suites as release gates in CI/CD pipelines.
• Experience with trajectory evaluation or simulation harnesses for multi-step, tool-using agents.
• Experience calibrating LLM-as-judge evaluators against human labels and measuring their agreement.
• Experience with permission-aware retrieval or context systems that track source provenance and freshness.
• Background in search relevance, recommendations, or ML evaluation, including offline and online metrics.

Responsibilities

• Own the roadmap and quality bar for agent evaluation across the platform, and influence architecture across teams.
• Build evaluation datasets and harnesses from production agent traces, including trajectory evaluation for multi-step runs.
• Integrate evaluations into deployment pipelines as release gates to catch regressions as models and prompts change.
• Calibrate LLM-as-judge scoring against human review and develop annotation workflows that route appropriate traces to reviewers.
• Run the learning loop by turning successful agent runs into reusable skills, with human-gated correction, replay, and drift monitoring.
• Own a permission-aware knowledge layer, including source ingestion with provenance and freshness and the context API that serves agents.
• Use agentic AI as a core development and coding tool.

Benefits

• Competitive pay.
• Employee discount.
• Financial savings and retirement resources.
• Support for physical and mental well-being.
• Leaves of absence and potential pay sources, subject to eligibility and applicable circumstances.
• Paid time off, with eligibility and amount depending on work location, employment status, salary or hourly status, and years of continued or bridged service.
• Certain roles may be eligible for incentive pay based on market norms and performance.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior