Content Automation Engineer

Team Red Dog • Redmond, WA

2d • Remote

About The Position

Team Red Dog is hiring a Content Automation Engineer for our client, a global productivity and collaboration leader and AI-driven technology organization. In this highly technical role, you'll help define how Microsoft 365 and Copilot evaluate AI-generated content by designing scalable automation frameworks, building intelligent testing systems, and developing human-in-the-loop quality strategies. You'll partner with engineering, content, and AI teams to create automated evaluation workflows that improve efficiency, strengthen quality standards, and directly influence experiences used by millions of customers worldwide. This is an outstanding opportunity for an automation-focused engineer who enjoys solving complex quality challenges at the intersection of AI, content, and software engineering.

Requirements

• Bachelor's degree in Computer Science, Engineering, Data Science, Information Systems, AI, or a related field.
• 2–4 years of experience in test automation, quality engineering, AI evaluation, or systems development.
• Strong Python programming skills and experience with automation tooling.
• Experience using Copilot, VSCode, or comparable AI development tools and infrastructure.
• Experience building testing frameworks, workflow automation solutions, or quality systems.
• Strong analytical skills with the ability to interpret data and translate findings into strategic recommendations.
• Excellent technical problem-solving and systems-thinking abilities.
• All applicants must be authorized to work in the U.S. without the need for sponsorship.

Nice To Haves

• Experience evaluating generative AI systems or LLM-powered products is preferred.
• Experience designing human evaluation programs or human-in-the-loop workflows is preferred.
• Familiarity with content quality measurement, content operations, or AI quality frameworks is a plus.
• Experience building internal tools, dashboards, or quality monitoring systems is preferred.
• Knowledge of Microsoft 365, Copilot, or productivity software ecosystems is desirable.

Responsibilities

• Design scalable testing and evaluation frameworks for templates, prompt templates, AI-generated content, and Microsoft 365 experiences.
• Define quality measurement approaches and establish testing standards across content ecosystems.
• Develop automated testing workflows, validation systems, and reusable quality frameworks.
• Build tools and processes that reduce manual effort while maintaining high-quality content standards.
• Identify opportunities to leverage AI for content validation, defect detection, and quality assurance.
• Design human-in-the-loop evaluation frameworks for assessing AI-generated outputs.
• Develop quality rubrics, benchmarking criteria, and repeatable testing methodologies.
• Analyze testing metrics and operational data to identify opportunities for continuous improvement.
• Measure automation effectiveness and optimize testing workflows over time.
• Partner with cross-functional teams to recommend automation strategies, evaluate quality risks, and drive adoption of scalable testing standards.

Benefits

• Health insurance (medical, dental, vision, and life)
• Employer-matched 401K plan
• Paid time off
• Paid holidays

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level