Backend Engineer - Monitoring

Sifflet

2d

About The Position

Sifflet implements data quality checks ranging from simple (detecting null values) to very complex (time series forecasting models to validate that the distribution of a set of columns hasn’t changed in an unexpected way, taking into account seasonality, one-off events…). This foundation powers many features, such as automatically merging related alerts into incidents. The monitoring team is responsible for evolving the data quality checks performed by Sifflet, and all associated workflows. As a member of this team, you will: Design and implement new types of data quality checks. Build features to allow users to efficiently monitor their entire data stack, such as automated monitor suggestions. Design advanced solutions to cut the alerting noise, such as automated incident root cause analysis. Scale our monitoring engine to support more and more customers. Some customers require monitoring massive data sets. Some projects you could be working on Add support for automated data profiling (understanding the expected distribution of values in all columns), and build an automated anomaly detection on top of that. Implement automated monitoring at scale, without requiring the user to manually define its own quality checks. Design new, more powerful data quality checks. Our stack The monitoring engine is built with Python 3 and its large data/ML ecosystem (notably PyTorch). Jobs are orchestrated with Temporal. The web API is written in (modern) Java with Spring Boot 3, the web frontend is a VueJS application written in Typescript. You may occasionally need to make minor changes to this code base. Infrastructure: Kubernetes (AWS EKS clusters), MySQL (on AWS RDS). Plus a few supporting services: Gitlab CI, Prometheus/Loki/Grafana, Sentry… While not directly part of our stack, expect to gain a lot of knowledge on many products in the modern data ecosystem. The subtleties of BigQuery or Snowflake will soon be very familiar to you.

Responsibilities

• Design and implement new types of data quality checks.
• Build features to allow users to efficiently monitor their entire data stack, such as automated monitor suggestions.
• Design advanced solutions to cut the alerting noise, such as automated incident root cause analysis.
• Scale our monitoring engine to support more and more customers.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed
• Number of Employees: 1-10 employees