REMOTE - Lead MLOps Engineer (Databricks)

State Farm•Dunwoody, GA

2d•Remote

About The Position

At the State Farm Applied Innovation Lab, we’re building an Agentic SDLC control plane, where GenAI helps with every part of how we build, test, deploy, and run software. We’re looking for someone who has run real workloads on Databricks in production and knows Delta Lake, Unity Catalog, and MLflow inside and out. Just as important, they already work the way we do, reaching for agentic tools naturally whether they’re building something new or tracking down a bug. If you’re happiest at the command line, you’ll feel right at home here. In this role, you’ll own the performance, quality, and production-readiness of select innovation projects running on Databricks – the ones where we dig deep to learn what excellence really looks like, then turn those lessons into patterns we can reuse on future projects. Other teams take care of the platform and infrastructure; you’ll own the layer above it: the data foundations, governance, FinOps, lineage, runbooks, and the handoff to Enterprise Technology. When a project leaves our hands, it’s fully governed, tagged at the project level, and optimized for cost, with a low L3 incident rate, complete runbooks, and self-healing wherever possible. This is a lead individual contributor role: you’ll set technical direction and raise the bar across teams without managing people. You’ll partner with teams across the enterprise, take on hard problems on Databricks, and help shape how we put the platform’s newest capabilities to work on the goals that matter most to State Farm. It’s a great fit if you want to grow your skills at the leading edge of Databricks and agentic AI, work on high-visibility initiatives, and do your best work in a fast-moving, experimental setting.

Requirements

Databricks (Non-Negotiable) – At least a year running Databricks in production at the application layer. You’ve designed and operated medallion architecture, and you know Delta Lake, Unity Catalog, and PySpark cold.
Production ML (Non-Negotiable) – You’ve kept models healthy in production, not just trained them and handed them off. You’ve been paged, written the RCA, and caught drift and silent degradation before they became someone else’s problem.
Agentic-Native (Non-Negotiable) – GenAI and agentic tools are already part of how you work every day, not an experiment. You’re at home on the command line and always looking for the next thing to automate.
MLflow 3.0 – Hands-on with the full lifecycle: registry, experiment tracking, deployment automation, retraining pipelines, and drift detection.
FinOps – You’ve tracked and optimized Databricks spend at the project level, and you treat compute budgets with the same rigor as SLAs.
Data Lineage & Observability – You’ve built or run systems that track lineage end to end and open tickets automatically on upstream breaks, schema changes, or SLA misses.
Production Handoff – You’ve prepared systems for operations teams with runbooks, self-healing patterns, incident targets, and governance checklists.
Systems Thinking – You see the whole system, not just the model or the pipeline. You’ve read Hidden Technical Debt in Machine Learning Systems, and you’ve lived it.
Engineering – Production-grade Python, PySpark, SQL, and AWS; CI/CD for data and ML pipelines; and testing strategies for non-deterministic systems.

Nice To Haves

Omnigent (Nice to Have) – Bonus points if you’ve worked with Omnigent, Databricks’ open-source agent meta-harness

Responsibilities

Data Foundations – Design and run medallion architecture (Bronze → Silver → Gold) with solid data quality, schema management, and Delta Lake tuning. Everything downstream depends on getting this right.
Application Performance – Hunt down bottlenecks in Spark queries, partitioning, caching, and Databricks cluster sizing, then fix them and keep them from coming back.
Unity Catalog & Lineage – Own schemas, access controls, and project-level tagging, with lineage-driven ticketing so upstream breaks surface right away.
MLOps / MLflow 3.0 – Take models through their full lifecycle, from experiment to production to retirement, including retraining and drift detection.
FinOps – Make Databricks cost visible by default: track, alert on, and optimize spend for every project.
ET Handoff – Define and hold the line on “done”: low L3 incident rate, complete runbooks, self-healing where possible, and governance and FinOps buttoned up. When a project crosses the handoff plane, it’s ready.
Patterns & Practices – Turn what you learn into templates and standards that make every future project better.
Agentic Integration – Bring agentic AI into everything you do: data validation, pipeline generation, testing, monitoring, documentation. If it can be augmented, it should be.
Technical Leadership & Collaboration – Set technical direction, mentor engineers, and pair with pod teams on architecture, code reviews, and debugging. You teach by doing.

Benefits

Compensation is based on our standard 38:45-hour work week!
Potential starting salary range: $100,000 - $200,000.
Potential yearly incentive pay up to 15% of base salary
Annual raise and bonus.
Robust health and wellbeing programs.
State Farm pays most of your healthcare premium.
Multiple healthcare plan options, including a high deductible plan.
All medical plans provide 100% coverage for in-network preventative care.
Access to vision, dental, telemedicine, 24/7 mental health professionals, and much more!
Educational benefits like industry leading training programs, top-notch tuition assistance programs, employee resource groups, and mentoring.
Fertility/IVF/adoption assistance
College coaching
National discount programs
Interactive monthly financial workshops
Free financial coaching
Generous time off policies
Opportunity to initially earn up to 20 days annually plus parental leave, paid holidays, celebration day, life leave (40 hours/year), bereavement leave, and community service/education support days.
Matching Gift Program
Good Neighbor Grant Program
Employee Assistance Fund
Free financial advisors
401(k) plan with company contributions of up to 7% of your salary.