Senior Software Engineer - Husky Storage
About The Position
As a Senior Software Engineer on the Husky team, you will work hands-on with a distributed storage system at petabyte scale. You will help evolve the platform to support the next level of scale required for the AI era. This involves migrating from a custom storage format to open standards like Parquet, Arrow, and DataFusion, enabling on-premise solutions, complex analytical/batch workloads, and powering AI observability. You will collaborate with Husky Ingestion, Metadata, and Platform teams, as well as Lakehouse on the Iceberg-compatible boundary. Additionally, you will engage with the open-source community to build scalable software solutions. Datadog fosters an engineering-driven, bottom-up culture with high agency and responsibility, promoting a blameless environment where challenges are overcome through mutual support. The company values remote work flexibility while also recognizing the benefits of in-person collaboration in its global offices.
Requirements
- Enthusiastic and experienced with large-scale distributed systems and/or storage engines
- Comfortable with improving and operating large distributed systems that customers rely on for critical business functions
- Strong fundamentals in columnar/file storage formats (Parquet, ORC, Arrow) or comparable systems
Nice To Haves
- experience with Go, Java, or Rust
- experience with compaction or compression/encoding schemes (RLE, dictionary, delta, gorilla)
- experience with object storage (S3/GCS/Azure Blob) at scale, including caching and cost optimization
- experience with Kafka and stream processing
- experience with Kubernetes
Responsibilities
- Drive Husky to the next level of adoption by migrating our storage format to open source Parquet and re-imagine compaction on a format-agnostic, DataFusion-based substrate
- Drive concrete cost-reduction work: block-level inverted index compression, combining the database with external objects at the row level, and storage tiering/caching
- Scale our solution to enable customers to manage event versioning at scale, using temporal-database techniques.
- Play an influential role in operating and deciding how to best operate our service going forward.
- Work across the write path (with Ingestion) and toward a cleaner Iceberg-based handoff to ADP/Lakehouse for batch/analytical workloads
Benefits
- New hire stock equity (RSUs)
- employee stock purchase plan (ESPP)
- Continuous professional development, product training, and career pathing
- Intradepartmental mentor and buddy program for in-house networking
- An inclusive company culture, access to Inclusion Talks
- Free, global mental health benefits for employees and dependents age 6+
- Competitive global benefits
- healthcare
- dental
- parental planning
- mental health benefits
- a 401(k) plan and match
- paid time off
- fitness reimbursements
- a discounted employee stock purchase plan
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
