Senior Data Platform Engineer

Saxon Global•Irving, TX

About The Position

We are seeking a world-class Data Platform Engineer to take complete operational and architectural ownership of one of the largest Apache Druid deployments in the industry. Our environment operates at extreme velocity and petabyte scale, processing real-time streaming data via Kafka across ~2,000 parallel ingestion tasks. If you have pushed Druid, Zookeeper, and the Linux kernel to their absolute breaking points—and know exactly how to tune them to stay up—this role is built for you. In this role, you will bridge the gap between deep infrastructure engineering, Unix Linux systems administration, and distributed big data platforms. You will design, scale, and automate massive Middle Manager/Indexer fleets, optimize deep storage, eliminate metadata DB bottlenecks, and implement bulletproof query isolation. Utilizing Ansible and Infrastructure-as-Code methodologies, you will ensure our clusters are highly available, self-healing, and meticulously monitored. If you are a platform engineer who thrives on solving complex concurrency, JVM Garbage Collection, and high-throughput streaming challenges at scale, join us to redefine what is possible with real-time analytics.

Requirements

Deep operational and architectural ownership of large-scale Apache Druid deployments.
Experience with extreme velocity and petabyte scale data processing.
Experience processing real-time streaming data via Kafka.
Experience tuning Druid, Zookeeper, and the Linux kernel.
Bridging the gap between deep infrastructure engineering, Unix Linux systems administration, and distributed big data platforms.
Designing, scaling, and automating Middle Manager/Indexer fleets.
Optimizing deep storage.
Eliminating metadata DB bottlenecks.
Implementing query isolation.
Utilizing Ansible and Infrastructure-as-Code methodologies.
Solving complex concurrency, JVM Garbage Collection, and high-throughput streaming challenges at scale.
Troubleshooting ingestion failures, stuck tasks, compaction issues, retention policies, and segment management problems.
Optimizing Druid queries, partitioning strategies, indexing specifications, and compaction configurations.
Configuring and maintaining Druid metadata stores using MySQL.
Creating dashboards, alerts, and monitoring solutions using Grafana, Prometheus, and logging platforms.
Working with platform teams to ensure high availability, disaster recovery, backup, and security compliance.
Participating in production support, incident response, root cause analysis, and performance tuning initiatives.
Collaborating with architects and business teams to understand analytical requirements and translate them into scalable solutions.

Responsibilities

Design, deploy, administer, and optimize large-scale Apache Druid clusters supporting PB-scale datasets.
Manage and troubleshoot Druid services.
Perform cluster upgrades, patching, capacity planning, and platform modernization activities.
Monitor cluster health, ingestion performance, query latency, segment distribution, and resource utilization.
Troubleshoot ingestion failures, stuck tasks, compaction issues, retention policies, and segment management problems.
Optimize Druid queries, partitioning strategies, indexing specifications, and compaction configurations.
Configure and maintain Druid metadata stores using MySQL.
Develop operational automation using Ansible and Infrastructure-as-Code methodologies.
Create dashboards, alerts, and monitoring solutions using Grafana, Prometheus, and logging platforms.
Work with platform teams to ensure high availability, disaster recovery, backup, and security compliance.
Participate in production support, incident response, root cause analysis, and performance tuning initiatives.
Collaborate with architects and business teams to understand analytical requirements and translate them into scalable solutions.