Lead Data Engineer

Yobi•New York, NY

52d•Hybrid

About The Position

As the Lead Data Engineer at Yobi, you will be at the center of how data moves, scales, and powers our Behavioral AI products. You will be working on various virtual teams and projects, with an eye towards schema modeling, orchestration, abstractions, workload efficiency, reliability/observability, and so on. This is a hands-on engineering role. What it takes to succeed in this role: You think in systems. You’re comfortable designing data workflows that balance clarity, modularity, and performance. You’re flexible and opinionated. You’re familiar with many different patterns for instrumenting data pipelines - push based, pull based, eventually consistent, 1 job per partition, batch, streaming, etc. and know when to reach for different patterns. You’re ready to influence and lead, aligning other flexible and opinionated engineers from other domains on the right conventions, abstractions, and strategy in an ambiguous and evolving environment. You have the hard skills. While we are big believers in “hire the person, not the specific language background”, our current data stack is Delta Lake, Scala Spark, Python Spark, and Airflow - it would be great if you have direct experience with at least some of these. You sweat the details. Schema evolution, data contracts, dependency graphs, metadata tracking — you know these are what make data reliable at scale. You’re pragmatic - you know when to polish and when to just ship. You care about performance. Wall clock time matters for us. Cost also matters for us. We think in many projects, there are opportunities to win on both, with the right data engineering. We prioritize attitude, culture, and general (technical) fit over matching perfectly into one of our job descriptions. If our mission and work resonates with you, we encourage you to apply. Tell us how you can help drive our products forward, even if you don’t feel like you are a perfect fit for some of the listings.

Requirements

You think in systems. You’re comfortable designing data workflows that balance clarity, modularity, and performance.
You’re flexible and opinionated. You’re familiar with many different patterns for instrumenting data pipelines - push based, pull based, eventually consistent, 1 job per partition, batch, streaming, etc. and know when to reach for different patterns.
You’re ready to influence and lead, aligning other flexible and opinionated engineers from other domains on the right conventions, abstractions, and strategy in an ambiguous and evolving environment.
You sweat the details. Schema evolution, data contracts, dependency graphs, metadata tracking — you know these are what make data reliable at scale.
You’re pragmatic - you know when to polish and when to just ship.
You care about performance. Wall clock time matters for us. Cost also matters for us.

Nice To Haves

our current data stack is Delta Lake, Scala Spark, Python Spark, and Airflow - it would be great if you have direct experience with at least some of these.