AI Engineer, Software Factory

Bernard•New York, NY

About The Position

Bernard runs scheduling, diagnostics, and support for appliance-repair service companies. We are a small team shipping a large surface, and we have decided that the way to keep doing that is to treat our own engineering as a factory with instrumentation: measured flow, measured quality, measured reliability, and agents doing a growing share of the bounded work inside a runtime that cannot hurt us. The factory has a second customer inside the company: our implementation teams. They sit with accounts, tune scheduling configuration, fix playbooks, and find the one roster or dial that explains a week of held tickets. Today the fastest fix is often a script with an apply flag and a careful person. The factory should hand them that power without the risk: changes they can propose from the field, simulate against the account's own history, ship behind a version-bound approval, watch under guardrails, and roll back by receipt rather than by memory. Every one of those field fixes should also become a measured treatment so we learn whether it worked and whether it belongs in the product. The same machinery points outward. Because we sit inside our customers' workflow, we see the numbers that matter to a service business: qualified intents to booked jobs, bookings to resolved cases, contribution per case, where the funnel leaks. Those numbers should be goals that agents work toward, with experiments, evidence, and a human holding the approval. Engineering metrics, field changes, and customer outcomes sit on one board and run through one loop. You will own that loop. The Software Factory board and its source adapters, the sandboxed agent runtime that produces changes with signed evidence, the authority and lease model around it, the measurement plane underneath, and the customer-facing goal and experiment runtime. This is an AI engineering role with product ownership, and most of your hardest problems will be about evidence, authority, and trust rather than prompts.

Requirements

Strong TypeScript on the backend. Our API is Fastify on Node, dashboards are Next.js, and a Python pipeline sits alongside. Postgres via Supabase holds metrics, experiments, and loop state. Neo4j holds the operational graph.
Developer-productivity instincts. You have measured an engineering org before, argued about what the numbers meant, and shipped the tooling or process change that moved them.
Empathy for the person on-site. You have been the one with production access at nine at night, and you want to build the thing that makes that safe for someone else.
Hands-on experience building agent systems in production: tool design, sandboxing, evaluation, cost control, and the discipline to keep an agent from inventing authority it was not granted.
An experimentation and causal mindset. You know the difference between a change event and an exposure, why assignment must be sticky, and why a guardrail breach pauses before anyone argues.
Respect for the boundary between built, connected, certified, and authorized. You do not collapse them into one green check, and you design systems that cannot.
Product ownership. You will talk to the people who run service businesses, to the engineers who sit with them, and to the engineers who run our platform, and you will ship for all three.

Nice To Haves

Platform, DevEx, or internal-tools experience, especially with GitHub, Linear, Sentry, or Grafana data.
Experience with feature-flag, rollout, or experimentation platforms.
Forward-deployed or solutions-engineering experience, on either side of the handoff.
Experience with Slack as a primary product surface.
Prior work in a services, logistics, or field-operations business.

Responsibilities

Own the Software Factory board. Source adapters for GitHub, Sentry, Grafana, and Linear normalize external facts into repository flow, review latency, change failure, incident load, and agent-factory health, on the same certified measurement plane as customer metrics.
Define the engineering metrics we actually steer by, version them, and name the ways each one can lie. Lead time means nothing if half the PRs are agent noise. You will know, and the board will say so.
Pick the bottlenecks worth an agent and the ones worth a human, and prove the difference with numbers rather than vibes.
Give field engineers a safe path from "I see the problem at this account" to "it is fixed and measured" without a raw database session. Configuration edits, roster and capability changes, playbook tweaks, and operational repairs become reversible, versioned treatment artifacts with the account as the scope.
Make simulation the default gate. A proposed change replays against the account's historical scheduling or the call-simulation harness before anyone can apply it, and the evidence travels with the approval request.
Build the operator tooling itself: dry-run by default, apply behind an explicit and expiring grant, compare-and-set writes, post-write verification, a receipt for every change, and rollback that works without anyone remembering what the old value was.
Turn field fixes into learning. When the same repair shows up across accounts, it should surface as a product hypothesis, not stay a tribal script.
A pinned, deny-by-default sandbox receives one compiled work envelope and one disposable worktree. It can edit and test inside the worktree and returns signed evidence. No network, no credentials, no ability to commit, push, open a PR, or touch production. You will decide what it is allowed to do next and prove it stays inside the lines.
The principal, grant, lease, budget, and episode model that governs every agent run, with Linear as the canonical work frontier and Slack as the interaction plane.
The evaluation harness: how we know an agent's change is good before a human reads it, and how we grow the share of work that lands with evidence attached.
Certified, versioned metric definitions with bounded scheduled materialization, saved observations, freshness, coverage, and calculation health.
The Big Board across three scopes: each customer project, the cross-project portfolio, and Bernard's own business and engineering. Thresholds, alert delivery to each account's channel, stewardship, and run health.
Build toward the north star and its honest components: profitable service yield, resolved-case yield, contribution per resolved case, with guardrails on customer effort, outcome quality, and cost to serve.
The experiment runtime: hypothesis, to immutable versioned treatment, to passed simulation, to version-bound approval, to apply, to durable sticky exposure, to evaluation against saved observations, to decision or rollback. Treatments today are action-rule rollouts and playbook canaries. You will add the next kinds, starting with the scheduling configuration our field teams touch most.
Turn Slack conversations with enterprise accounts into classified signals, feature requests, and candidate goals, and get the right ones in front of the right human.
Help an account's agents pursue its goals: diagnose why a metric moved, propose a hypothesis, draft a reversible treatment, cite the observations that support a verdict, while the human records the decision.
Make the chain legible to a customer. What was measured, what changed, who was exposed, what improved, what we learned.