AI Engineer, Software Factory

Bernard•New York City, NY

10h

About The Position

Bernard is hiring an AI Engineer to own its Software Factory: an instrumented engineering and product system for improving developer productivity, safely enabling implementation teams to make account-specific changes, and helping agents work toward customer outcomes. The role combines AI engineering with product ownership. Its central concerns are evidence, authority, and trust—not prompts. The engineer will own a Software Factory board and its source adapters; a sandboxed agent runtime that produces signed evidence; the authority, grant, and lease model governing that runtime; the measurement plane; and customer-facing goal and experiment workflows. The system will connect engineering metrics, field changes, and customer outcomes in a single measured loop. For implementation teams, the role will provide a safe workflow for proposing account-scoped configuration, roster, capability, playbook, and operational changes; simulating them against historical data or call simulations; approving and applying version-bound treatments; monitoring them under guardrails; and rolling them back with durable receipts. Field fixes should also generate measurable learning and product hypotheses. The agent runtime will operate in a pinned, deny-by-default sandbox with a compiled work envelope and disposable worktree. Agents may edit and test within that worktree and return signed evidence, but must not have network access, credentials, or the ability to commit, push, open pull requests, or affect production. The role also owns the evaluation harness and the model of principals, grants, leases, budgets, and episodes that governs agent runs. The measurement and experimentation systems will support certified, versioned metrics; bounded materialization; saved observations; freshness, coverage, and calculation-health monitoring; and a Big Board spanning customer projects, the portfolio, and Bernard’s business and engineering. Customer-facing experiments will progress from hypothesis through immutable treatment, simulation, version-bound approval, application, sticky exposure, evaluation, and a decision or rollback. The engineer will also help agents interpret customer signals and metrics, propose evidence-backed reversible treatments, and make the resulting measurement and decisions legible to customers.

Requirements

Strong backend TypeScript experience.
Experience measuring an engineering organization, interpreting its metrics, and shipping tooling or process changes that improved them.
Empathy for people working on-site and an interest in making production-affecting work safer.
Hands-on experience building production agent systems, including tool design, sandboxing, evaluation, cost control, and authority boundaries.
An experimentation and causal mindset, including understanding the distinction between a change event and an exposure, sticky assignment, and pausing on guardrail breaches.
Respect for the distinctions between systems or capabilities that are built, connected, certified, and authorized; ability to design systems that preserve those boundaries.
Product ownership skills and willingness to work with service-business operators, implementation engineers, and platform engineers.

Nice To Haves

Platform, developer-experience, or internal-tools experience, especially involving GitHub, Linear, Sentry, or Grafana data.
Experience with feature-flag, rollout, or experimentation platforms.
Forward-deployed or solutions-engineering experience on either side of the handoff.
Experience using Slack as a primary product surface.
Prior experience in a services, logistics, or field-operations business.

Responsibilities

Own the Software Factory board and source adapters for GitHub, Sentry, Grafana, and Linear, normalizing external facts into engineering and agent-factory metrics.
Define, version, and qualify engineering metrics, including identifying ways those metrics can be misleading.
Identify work suited to agents versus people and measure the results.
Build safe operator tooling for field engineers to propose account-scoped changes without raw database access.
Make simulation the default gate for proposed changes, replaying them against account history or the call-simulation harness and attaching evidence to approval requests.
Implement dry-run defaults, explicit expiring grants, compare-and-set writes, post-write verification, change receipts, and reliable rollback.
Turn recurring field fixes into measurable product hypotheses.
Own a pinned, deny-by-default agent sandbox and determine what agents may do next while demonstrating that they remain within their authority.
Own the principal, grant, lease, budget, and episode model governing agent runs, using Linear as the canonical work frontier and Slack as the interaction plane.
Build the evaluation harness for agent-generated changes and increase the proportion of work delivered with evidence.
Own certified, versioned metric definitions and their scheduled materialization, observations, freshness, coverage, and calculation-health tracking.
Build and operate the Big Board across customer projects, the cross-project portfolio, and Bernard’s business and engineering; manage thresholds, alerts, stewardship, and run health.
Advance metrics for profitable service yield, resolved-case yield, and contribution per resolved case, with guardrails for customer effort, outcome quality, and cost to serve.
Own the customer experiment runtime from hypothesis and treatment through simulation, approval, application, exposure, evaluation, decision, or rollback.
Extend treatments beyond action-rule rollouts and playbook canaries, beginning with scheduling configuration used by field teams.
Turn enterprise-account Slack conversations into classified signals, feature requests, and candidate goals, and route them to appropriate human decision-makers.
Help account agents diagnose metric changes, propose hypotheses, draft reversible treatments, cite supporting observations, and leave decisions to humans.
Make customer-facing experiment chains legible, including what was measured, what changed, who was exposed, what improved, and what was learned.
Work with service-business operators, implementation engineers, and platform engineers to ship products that serve all three groups.