Staff Engineer, Applied AI & Build Standards

Best Buy • Richfield, MN

5d • $121,050 - $216,650 • Hybrid

About The Position

As a Staff Engineer on Best Buy’s Applied AI & Build Standards team, you will develop evaluation frameworks, quality standards, and release controls to help ensure AI agents perform safely and effectively. Working across the platform, you will help turn successful agent interactions into reusable capabilities and trusted organizational knowledge. The team, known internally as AI Rails, builds a shared platform and standards that enable Best Buy engineers and technologists to use AI agents with appropriate access, security, and cost controls.

The role is hybrid. You are required to work on site Monday through Thursday at Best Buy’s corporate office in Richfield, Minnesota, and may work virtually from home or another non-Best Buy location on Friday. The role is eligible for immigration sponsorship. Use of Agentic AI as a core development and coding tool is part of the role.

Requirements

• At least 5 years of relevant experience with a bachelor’s degree, or equivalent relevant professional experience.
• At least 3 years of professional software development experience in Python.
• At least 2 years of experience building production data pipelines or search and retrieval systems, including data quality checks.
• At least 1 year of experience with formal or informal leadership.
• Hands-on experience building evaluation frameworks for LLM-based applications or AI agents.
• Daily, hands-on use of AI coding agents to write, test, and review code.

Nice To Haves

• Bachelor’s degree in IT, Computer Science, Engineering, or a related field.
• At least 7 years of relevant professional experience.
• Experience running evaluation suites as release gates in CI/CD pipelines.
• Experience with trajectory evaluation or simulation harnesses for multi-step, tool-using agents.
• Experience calibrating LLM-as-judge evaluators against human labels and measuring agreement.
• Experience with permission-aware retrieval or context systems that track source provenance and freshness.
• Background in search relevance, recommendations, or ML evaluation, including offline and online metrics.

Responsibilities

• Own the roadmap and quality bar for agent evaluation across the platform, and influence architecture across teams.
• Build evaluation datasets and harnesses from production agent traces, including trajectory evaluation for multi-step runs.
• Integrate evaluations into deployment pipelines as release gates to catch regressions as models and prompts change.
• Calibrate LLM-as-judge scoring against human review and create annotation workflows that route appropriate traces to reviewers.
• Turn successful agent runs into reusable skills through a learning loop with human-gated correction, replay, and drift monitoring.
• Own a permission-aware knowledge layer, including source ingestion with provenance and freshness and the context API that serves agents.

Benefits

• Competitive pay.
• Employee discount.
• Financial savings and retirement resources.
• Support for physical and mental well-being.
• Leaves of absence and potential pay sources based on eligibility and applicable circumstances.
• Paid time off for eligible full-time and part-time employees, subject to location, employment status, salary or hourly status, and years of service.
• Certain roles may be eligible for incentive pay.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior