Lead Engineer, Data Processing (Databricks)

CargoSprint

4d•Remote

About The Position

You're a hands-on leader who can run a team and still ship. You care about operational reliability, clear ownership boundaries, and measurable throughput. You modernize systems safely, reduce toil, and build habits that make pipelines dependable. This role leads our Data Processing team. You'll own the pipelines and processing jobs that transform data for product and business use cases, with an emphasis on modernization, reliability, and data quality. Strong experience with Databricks (or similar Spark-based platforms) is required.

Requirements

8+ years building production systems; 2+ years leading engineers or acting as a team tech lead.
Strong experience with data processing systems (ETL/ELT, streaming, batch processing, orchestration).
Hands-on experience with Databricks and Spark (PySpark/Scala) and strong SQL.
Experience modernizing legacy systems safely and incrementally.
Operational discipline: monitoring, alerting, incident response, and root cause analysis for pipelines.
Ability to drive cross-team alignment and translate requirements into reliable delivery.

Nice To Haves

Lakehouse patterns (Delta Lake, Iceberg, or Hudi) and data catalog/governance tools.
.Net or Java exprertise
Event-driven architectures (Kafka/Kinesis/PubSub) and streaming pipelines.
FinTech, payments, billing, invoicing, or other high-volume transactional domains.
Experience enabling self-serve analytics for business users.
Logistics, cargo, or supply chain experience.
Spanish language proficiency.

Responsibilities

Lead the Data Processing team: technical direction, delivery cadence, coaching, and hiring support (as needed).
Own core processing systems end-to-end: design, implementation, quality, observability, and on-call readiness.
Drive modernization of legacy processing: simplify flows, reduce failure modes, improve performance and cost.
Build and operate pipelines (batch and/or streaming) with strong data quality, lineage, and backfill strategies.
Establish SLAs/SLOs for key pipelines and improve monitoring, alerting, and incident response.
Partner with application engineering, finance, and analytics to prioritize work and deliver dependable outputs.
Raise the engineering bar via standards, PR reviews, runbooks, and pragmatic architecture decisions.