Staff Engineer, Data Platform

DRW•Chicago, IL

4d•$200,000 - $275,000

About The Position

DRW’s Unified Platform organization builds managed data services, shared infrastructure, and metadata management systems that support its common query layer and batch and streaming pipelines. The Staff Engineer, Data Platform will help shape and uphold the firm’s long-term data platform vision, translating data strategy into engineering products and managed services used by Traders, Quantitative Researchers, and Back-Office personnel. This individual-contributor role combines strategic thinking, architectural leadership, software development, and mentorship. The engineer will work across tooling, producer, and consumer teams to connect business priorities with technical direction and hands-on delivery, and will lead technical initiatives spanning multiple teams.

Requirements

Extensive experience designing, building, and operating large-scale, data-intensive distributed systems, including technical leadership of initiatives spanning multiple teams.
Experience coordinating with engineering leadership to accelerate team- and organization-wide delivery within existing processes and practices.
A track record of translating data or business strategy into engineering products and technical roadmaps, aligning near-term delivery with a long-term platform vision.
A track record of building and offering managed platform services to other teams, owning provisioning, upgrades, reliability, and ongoing support throughout the service lifecycle, and incorporating user requirements and feedback into product and roadmap decisions.
Experience scaling metadata management systems, such as data catalogs or lineage tracking systems, across multiple storage and compute technologies.
Expertise in Java and/or Python, with a proven ability to deliver high-quality, maintainable production software; strong familiarity with SQL and Linux environments.
Deep understanding of distributed systems trade-offs, including consistency, fault tolerance, concurrency, partitioning, and recovery from partial failures.
Production experience building batch and stream processing systems at scale, including practical knowledge of state management, replay, backpressure, and delivery guarantees.
Experience deploying data processing and query technologies such as Flink, Spark, and ClickHouse, and ability to guide platform users on their performance and operational trade-offs.
Experience with multiple storage technologies, including object storage, relational or NoSQL databases, and event streaming systems such as Kafka; in-depth knowledge of open table format internals such as Iceberg or Delta Lake.
Extensive experience with data modeling, schema governance and evolution, and formats and serialization systems such as Arrow, Parquet, Protobuf, or Avro.
Familiarity with building and operating cloud-native data infrastructure, including Kubernetes, workflow orchestration, automated deployment, and production monitoring.
Excellent written and verbal communication skills, including communicating a technical vision, explaining trade-offs, aligning stakeholders, and leading through influence across a globally distributed organization.
Strong technical judgment and problem-solving skills, including prioritizing competing needs, delivering incrementally in a fast-paced trading environment, and developing other engineers.

Nice To Haves

Experience scaling managed data services across teams with different workloads, service expectations, and operational constraints.
Hands-on knowledge of data governance in regulated environments.
Familiarity with financial datasets and the needs of trading, quantitative research, or back-office users.

Responsibilities

Contribute to and uphold the vision for DRW’s Data Platform, guiding its architecture and evolution in support of the firm’s data strategy.
Work with engineering leads, product managers, and stakeholders to understand business priorities, translate data strategy into engineering products and platform capabilities, and contribute to a sequenced technical roadmap.
Design, guide, and build managed services for data storage, distributed query execution, batch and streaming processing, and workload orchestration; own their lifecycle from provisioning and upgrades through production support, and refine offerings using user feedback.
Lead delivery across team boundaries, translating ambiguous requirements into practical designs and measurable outcomes while balancing immediate needs with long-term platform goals.
Drive architecture reviews and technology decisions, document trade-offs, and build consensus around patterns that support the platform vision.
Own the strategy for large-scale data governance and build capabilities for metadata management, discovery, lineage, schema evolution, and data quality; partner with security and data stakeholders on access controls and usage tracking.
Troubleshoot distributed systems, contribute to production support and incident response, and drive lasting improvements through root-cause analysis, automation, and operational documentation.
Mentor engineers across teams through design reviews, code reviews, and hands-on collaboration, strengthening engineering practices and helping others take ownership of complex systems.