Senior SW Engineer, Data Infrastructure

Roberts Recruiting • Cambridge, MA

4d

About The Position

The Data Infrastructure team is responsible for the various datastores we use, including MySQL, HBase, ElasticSearch, and Kafka. We aim to provide four nines (or better) of reliability for all of these, as well as to help our developers use them easily and safely. We spend a lot of time building automation, tooling, and monitoring to equip developers to understand and optimize their datastore usage and minimize the impact of any operational issues on customers and developers alike. We’re also operationally responsible for a huge volume of traffic to and from these datastores. Our HBase clusters serve over 3 million requests/second across 220+ tables, while our ElasticSearch clusters serve over 20k searches/second and 50k indexes/second to 90+ billion documents. Streaming that data to and from applications amounts to more than 3 GB/sec of data through our Kafka clusters, with hundreds of producers and consumers.

Requirements

• Experienced with some or most of our datastores (MySQL, HBase, ElasticSearch, Kafka) as a user and/or operationally.
• Excited to build great tools for developers so they can work faster and more reliably.
• Interested in working with and supporting infrastructure at scale.
• Eager to learn and grow.

Responsibilities

• Provide four nines (or better) of reliability for datastores (MySQL, HBase, ElasticSearch, Kafka).
• Build automation, tooling, and monitoring to help developers use datastores easily and safely.
• Equip developers to understand and optimize their datastore usage.
• Minimize the impact of operational issues on customers and developers.
• Manage and operate a huge volume of traffic to and from datastores.
• Support HBase clusters serving over 3 million requests/second across 220+ tables.
• Support ElasticSearch clusters serving over 20k searches/second and 50k indexes/second to 90+ billion documents.
• Manage data streaming through Kafka clusters, handling more than 3 GB/sec of data with hundreds of producers and consumers.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed