Senior Data Engineer - Streaming

The Coca-Cola Company•Atlanta, GA

2d•$152,000 - $178,300

About The Position

The Senior Data Engineer – Streaming will evolve Coca-Cola’s enterprise data estate from batch-reliant processing to a continuous, event-driven ecosystem. The role focuses on designing, standardizing, and codifying streaming and Kappa Architecture patterns while minimizing redundant batch layers. Rather than only building isolated data pipelines, this engineer will architect foundational frameworks, reference implementations, and low-code configurations that allow engineering teams across the organization to deploy robust real-time data streams. The target platform is Microsoft Fabric, using Spark Streaming and/or Fabric Real-Time Intelligence. The ideal candidate champions log-centric data design and has deep expertise in enterprise Apache Kafka environments. The role also serves as an internal authority on streaming design patterns across Kafka and Event Hubs, and includes creating standardized stream-health dashboards. The position is based in the United States, with Atlanta listed as a city. Travel is 0%–25%; relocation is not provided. Coca-Cola does not offer employment-based visa sponsorship for this position. Applicants must be authorized to work full-time in the United States without requiring Coca-Cola sponsorship.

Requirements

At least 8 years of experience in data platform architecture, including at least 3 years architecting and operating production-grade Streaming and Kappa Architectures.
At least 4 years of experience running Spark Streaming and Apache Kafka pipelines.
Understanding of Kafka topic partitioning strategies, consumer group mechanics, Avro/JSON schema registries, and high-throughput stream performance tuning.
Deep familiarity with stream-processing engines such as Spark Structured Streaming, Kafka Streams, or Apache Flink.
Strong Scala and/or PySpark experience with stateful Structured Streaming.
Advanced knowledge of Delta Lake internals, particularly append-only streaming sinks and checkpointing behaviors.
A portfolio demonstrating developer-facing frameworks, custom connectors, or configuration-driven streaming engines that abstract infrastructure.
Ability to explain complex streaming topologies and real-time concepts to application developers and executive stakeholders.
Currently authorized to work in the United States on a full-time basis without requiring Coca-Cola sponsorship to continue working legally.

Nice To Haves

Confluent Certified Developer or Kafka Administrator certification.
Experience modernizing Lambda architectures toward Kappa-first designs.
Hands-on experience with Fabric Real-Time Intelligence, including Fabric Eventstreams and KQL Databases; equivalent deep experience with Azure Event Hubs, Stream Analytics, or Databricks Delta Live is also relevant.

Responsibilities

Build a reusable enterprise streaming architecture playbook focused on Kappa-style patterns, event-driven processing, and near-real-time analytics on Fabric.
Architect, document, and templatize standardized replay and backfill mechanisms, including offset resets, checkpoint and state management, and sink rebuild and deduplication strategies.
Create distinct architectural blueprints and guardrails for sub-second real-time and near-real-time business service-level agreements.
Package stateful processing, watermarking, late-arriving data remediation, and complex windowing into reusable templates.
Codify design patterns for effectively-once processing and end-to-end idempotency with deterministic replay.
Build metadata-driven frameworks that handle upstream schema drift without interrupting or crashing continuously running downstream streaming queries.
Serve as the internal authority on streaming design patterns across Kafka and Event Hubs, and design scalable ingestion into Fabric.
Build standardized dashboards to track end-to-end data lag, processing throughput, backpressure, and poison-pill routing through Dead Letter Queues (DLQ).