Senior Data Scientist (Agentic UX), Design

Sitetracker•Alpena, MI

8d•$145,000 - $215,000

About The Position

As a Senior Data Scientist on Sitetracker’s Design team, you will shape agentic experiences for critical infrastructure companies. The role focuses on defining, measuring, and improving the quality of agent behavior across real-world scenarios, connecting rigorous experimentation and evaluation to customer and commercial outcomes. You will be embedded in a scrum team, working day to day with Engineering and Design, owning decisions for your experience area, mentoring others, and shipping work to production. You will investigate customer and product behavior, study the market and relevant technologies, and design full agent experiences across web and mobile. This includes architecting Scout graphs, context retrieval, decision paths, and communication patterns, as well as building code, data pipelines, instrumentation, prototypes, and evaluation harnesses. You will define and track quality across dimensions such as correctness, grounding, safety, refusal, latency, and tone; design offline and online evaluations and experiments; and use customer feedback and adoption data to iteratively improve experiences. The role also involves mentoring teammates, reviewing their work, sharing work for feedback, presenting technical and quantitative concepts to varied audiences, and influencing product direction with evidence-backed recommendations. The position is intended for a hands-on builder who is comfortable with ambiguity, model behavior, enterprise workflows, and close collaboration with customers and engineering teams.

Requirements

Demonstrated statistical and machine-learning fluency applied to complex production behavior, including distinguishing signal from noise and root causes from symptoms.
Ability to define quality for agentic experiences and translate it into robust metrics, evaluations, and datasets.
Ability to design and analyze experiments, including A/B tests, holdouts, and shadow runs, with appropriate sizing and sound conclusions.
Experience or capability building offline and online LLM evaluation systems across correctness, grounding, safety, tone, and latency.
Ability to build and ship production code, functional prototypes, pipelines, instrumentation, and evaluation harnesses within an existing codebase.
Fluency with AI-assisted development tools and a willingness to make building a primary way of working.
Technical understanding sufficient to discuss platform architecture, retrieval, grounding, engineering tradeoffs, graph logic, and AI behavior as a peer.
Ability to design context architecture, decision paths, and communication patterns for agent behavior across a broad range of inputs.
Ability to turn complex enterprise workflows into usable agent experiences and connect design decisions to commercial outcomes.
Ability to work directly with customers, run pilots, understand workflows, and adapt work based on customer behavior and needs.
Ability to take ownership of ambiguous challenges, make independent decisions, prioritize, and deliver at pace.
Strong product and industry learning skills, including understanding customer workflows and what constitutes a correct answer in those contexts.
Ability to present compellingly to senior stakeholders and translate technical and quantitative concepts for diverse audiences.
Ability to mentor others and provide substantive, actionable feedback on evaluations, experiments, analyses, and agent designs.

Nice To Haves

Experience or familiarity with telecommunications, EV charging, renewables, utilities, or other critical infrastructure industries.
Experience with Salesforce ISV products, SLDS-based web components, or purpose-built mobile experiences.
Experience with Scout, Watch, or architecting agent graphs and sophisticated AI interactions.
Experience maintaining competitive and industry awareness across multiple verticals and contributing substantive observations to team channels.
Experience mentoring or managing an intern.

Responsibilities

Own day-to-day decisions for an experience area, define leading indicators, scope and prioritize work, and make tradeoffs across user needs, agent behavior, technical constraints, and business goals.
Study the competitive landscape, product and AI capabilities, and the critical infrastructure industries served by Sitetracker; connect findings to product and behavior decisions.
Investigate cross-team problems using logs, traces, evaluations, and customer behavior; instrument gaps, analyze root causes, and turn findings into action.
Work directly with customers, run pilots, study product usage, and translate behavioral patterns into design decisions.
Design agent experiences across the full distribution of inputs, including context architecture, decision paths, communication patterns, product surfaces, and behavior contracts.
Build and ship production code, prototypes, data pipelines, instrumentation, and evaluation harnesses using AI-assisted development tools and the existing codebase.
Architect Scout graphs and AI interactions, and define Scout’s decision paths, context architecture, communication patterns, and product surfaces.
Create and maintain comprehensive evaluations and datasets; measure correctness, grounding, safety, refusal, latency, and tone, and track quality over time in Watch.
Design offline and online evaluation strategies, A/B tests, holdouts, and shadow evaluations; size experiments appropriately and draw evidence-based conclusions.
Conduct usability and behavior reviews, build prototypes to test Scout behavior, and run customer betas and previews.
Establish metrics tied to commercial outcomes, track adoption and results, and iterate on shipped features to improve performance.
Build dashboards and analyses that make adoption and outcomes visible across the organization.
Mentor colleagues, review evaluations, experiments, analyses, and agent designs, and provide actionable feedback that raises quality standards.
Share work early and often, invite critique, contribute to technical and industry discussions, and communicate complex technical and quantitative concepts to diverse audiences.
Present recommendations to senior stakeholders and influence roadmaps with evidence-backed positions.
Potentially manage a dedicated intern, setting expectations and ensuring meaningful work.