Database Engineer

Ahrefs • Palo Alto, CA

1d • Onsite

About The Position

Ahrefs is hiring a Database Engineer to build and operate the data layer supporting one of the web’s largest crawlers. The Database team owns databases and storage end to end, from source code through production. Its infrastructure includes ClickHouse, MooseFS, SeaweedFS, TiDB and Elasticsearch, running on about 1,200 servers and holding hundreds of petabytes across data centres in Singapore, the US and the UK. When systems need changes, the team modifies their source code, validates changes on production data and rolls them out directly.

The engineer will develop and operate these database and storage systems, write and ship code, and support it in production. The role also involves working with AI coding agents as a normal part of the job. The position is on-site in Singapore, Tokyo, Palo Alto, California, or Cambridge, Massachusetts; candidates need the right to work in the office they choose.

Requirements

• Strong C++, C and Rust programming skills, ability to work in Go, and comfort working in codebases of millions of lines.
• Deep knowledge of database storage engines, including merges or compaction, replication logs and query execution.
• Understanding of distributed consensus, such as Raft or Paxos, and system failure scenarios.
• Knowledge of Linux storage internals, including page cache, fsync, FUSE, file systems, and NVMe and HDD behavior.
• Experience running databases or storage on hundreds of nodes and leading incident recovery.
• Ability to debug with gdb, perf and sanitizers.
• Habit of measuring changes on production data before determining their impact.

Nice To Haves

• Contributions to an open-source database or file system.
• OCaml experience.
• Puppet or similar configuration management experience.

Responsibilities

• Build storage and query features in ClickHouse, including replication over a shared file system and storage policies that place data according to merge history.
• Keep up with upstream ClickHouse LTS releases, maintain green CI, and regularly ship Ahrefs releases.
• Improve the availability and durability of MooseFS and SeaweedFS, including Raft-based metadata servers, follower reads, an intent log and automatic re-replication.
• Identify production performance problems and fix them in code, from query plans to FUSE clients.
• Trace data corruption to its root cause, including issues in compression, memory allocation or the compiler, and repair damaged data.
• Roll out releases and storage migrations across about 1,200 servers in three regions without downtime.
• Run and tune TiDB, TiKV, TiCDC and Elasticsearch, including scheduling, replication lag, backups and disk balance.
• Validate risky designs with TLA+ and fault injection, and automate recurring work with watchdogs, balancers and dashboards.
• Build an SRE agent that monitors database clusters, diagnoses incidents and proposes fixes.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed