Senior Software Engineer - Husky Storage
About The Position
Husky is Datadog’s distributed, petabyte-scale columnar event store, at the heart of its Event Platform and powering products including Logs, RUM, APM, Cloud Network Monitoring, and Netflow. It was built to store and query massive volumes of event data at low cost, with data fully queryable within seconds of arrival. As a Senior Software Engineer on the Husky Storage team, you will work hands-on with a distributed storage system at petabyte scale and help evolve it for the next level of scale. The team is moving from a custom storage format toward open standards such as Parquet, Arrow, and DataFusion. This work aims to enable on-premises solutions, increasingly complex analytical and batch workloads, and AI observability. You will partner with Husky Ingestion, Metadata, and Platform, and increasingly with Lakehouse on the Iceberg-compatible boundary between Husky and analytical or batch workloads. You will also work with the open-source community to build reusable software solutions that can be operated at scale. Datadog describes its culture as engineering-driven, bottom-up, and high-agency, with employees empowered to shape the company’s technological and strategic direction. The company emphasizes blameless collaboration, open source, and flexibility in working remotely while also valuing its offices and bringing people together.
Requirements
- Enthusiasm and experience with large-scale distributed systems and/or storage engines.
- Comfort improving and operating large distributed systems that customers rely on for critical business functions.
- Strong fundamentals in columnar/file storage formats such as Parquet, ORC, or Arrow, or comparable systems.
Nice To Haves
- Experience with Go, Java, or Rust.
- Experience with compaction or compression/encoding schemes such as RLE, dictionary, delta, or gorilla.
- Experience with object storage such as S3, GCS, or Azure Blob at scale, including caching and cost optimization.
- Experience with Kafka and stream processing.
- Experience with Kubernetes.
Responsibilities
- Drive adoption of Husky by migrating its storage format to open-source Parquet and reimagining compaction on a format-agnostic, DataFusion-based substrate.
- Deliver cost-reduction work, including block-level inverted-index compression, combining the database with external objects at the row level, and storage tiering and caching.
- Scale the solution to enable customers to manage event versioning at scale using temporal-database techniques.
- Play an influential role in operating the service and deciding how it should be operated going forward.
- Work across the write path with Ingestion and advance a cleaner Iceberg-based handoff to ADP/Lakehouse for batch and analytical workloads.
- Collaborate with the open-source community to build reusable software solutions that can be operated at scale.
Benefits
- New-hire stock equity (RSUs) and employee stock purchase plan (ESPP).
- Continuous professional development, product training, and career pathing.
- Intradepartmental mentor and buddy program for in-house networking.
- Inclusive company culture and ability to join Community Guilds (Datadog employee resource groups).
- Free global mental health benefits for employees and dependents age 6 and older.
- Healthcare, dental, parental planning, and mental health benefits.
- 401(k) plan and match.
- Paid time off.
- Fitness reimbursements.
- Discounted employee stock purchase plan.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
