Senior Software Engineer - Husky Storage

Datadog • Boston, MA

1d

About The Position

As a Senior Software Engineer on Datadog’s Husky Storage team, you will work hands-on with Husky, a distributed, petabyte-scale columnar event store at the heart of the Event Platform. Husky stores and queries massive volumes of event data at low cost, with data queryable within seconds of arrival, and supports products including Logs, RUM, APM, Cloud Network Monitoring, and Netflow.

You will help evolve Husky to support the next level of scale and AI observability. The team is moving from a custom storage format toward open standards such as Parquet, Arrow, and DataFusion, enabling on-premises solutions and increasingly complex analytical and batch workloads. You will partner with Husky Ingestion, Metadata, and Platform teams, and work increasingly with Lakehouse on the Iceberg-compatible boundary between Husky and analytical or batch workloads. You will also work with the open source community to build reusable software solutions that can be operated at scale.

Datadog describes its culture as engineering-driven, bottom-up, and high-agency, with employees empowered to influence the company’s technological and strategic direction. The company emphasizes blameless collaboration, open source, and flexibility to work remotely while also valuing in-person collaboration at its offices worldwide.

Requirements

• Experience with large-scale distributed systems and/or storage engines.
• Comfort improving and operating large distributed systems that customers rely on for critical business functions.
• Strong fundamentals in columnar or file storage formats such as Parquet, ORC, or Arrow, or comparable systems.

Nice To Haves

• Experience with Go, Java, or Rust.
• Experience with compaction or compression/encoding schemes such as RLE, dictionary, delta, or gorilla.
• Experience with object storage such as S3, GCS, or Azure Blob at scale, including caching and cost optimization.
• Experience with Kafka and stream processing.
• Experience with Kubernetes.

Responsibilities

• Drive adoption of Husky by migrating its storage format to open-source Parquet and reimagining compaction on a format-agnostic, DataFusion-based substrate.
• Deliver cost-reduction work, including block-level inverted-index compression, combining the database with external objects at the row level, and storage tiering and caching.
• Scale the solution so customers can manage event versioning at scale using temporal-database techniques.
• Help operate the service and influence decisions about how it should be operated going forward.
• Work across the write path with Ingestion and toward a cleaner Iceberg-based handoff to ADP/Lakehouse for batch and analytical workloads.
• Collaborate with the Husky Ingestion, Metadata, Platform, and Lakehouse teams, as well as the open source community.

Benefits

• New-hire stock equity (RSUs) and employee stock purchase plan (ESPP).
• Continuous professional development, product training, and career pathing.
• Intradepartmental mentor and buddy program for in-house networking.
• Inclusive company culture and the ability to join Community Guilds (Datadog employee resource groups).
• Free global mental health benefits for employees and dependents age 6+.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed