Senior Software Engineer, Big Data, Tech Enablement

ZipRecruiter•Santa Monica, CA

2d•$190,000 - $250,000•Hybrid

About The Position

The Senior Software Engineer will join ZipRecruiter’s Tech Enablement team, which builds scalable, reliable, and efficient infrastructure, tools, and processes that help engineering teams deliver products. The role focuses on data infrastructure and systems that enable engineers and data scientists to work with data and machine-learning technologies, as well as AI tools that support data exploration and analysis. The engineer will design, build, deploy, and maintain data ingestion pipelines and datalake systems that make web events available for downstream applications. They will work with large, evolving datasets using batch and streaming approaches, and develop MLOps frameworks to improve the velocity, quality, observability, and reliability of machine-learning development and products. The role also includes building systems and tools for data documentation and contributing to Sift, an agentic data exploration and analysis tool intended to improve access to data and accelerate data-driven decisions.

Requirements

At least 4 years of professional software development experience, including building and operating data pipelines that feed into a customer-facing product.
Hands-on Apache Spark experience writing, tuning, and debugging batch and/or streaming jobs at scale.
Experience deploying and operating containerized services and batch jobs.
Solid computer science fundamentals, including data structures, algorithms, distributed systems, and testing.
Professional software development experience, including building and operating data pipelines and infrastructure that feed into a customer-facing product.

Nice To Haves

At least 4 years of professional software development experience focused on data-intensive systems.
Working knowledge of AWS, including S3, IAM, EC2, and MSK or equivalents, and experience instrumenting jobs with metrics, logs, and alerts.
Experience with a data lake stack, including Parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake.
Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.
Experience with a workflow orchestrator such as Apache Airflow, including authoring DAGs, backfills, dependency management, and on-call support for scheduled pipelines.
Experience with open table format internals, including schema evolution, partitioning, and compaction.
Experience deploying services on Kubernetes with Docker.
Experience building and maintaining a data platform that powers a user-facing product, including infrastructure and tools for streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery.
BS, MS, or PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.

Responsibilities

Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and through event streams for downstream applications.
Process large and evolving datasets using batch and streaming techniques, balancing speed, efficiency, and reliability.
Design, build, deploy, and maintain MLOps frameworks and systems that improve the velocity, quality, observability, and reliability of machine-learning development and products.
Develop frameworks and systems that improve the comprehensiveness and quality of data documentation, including tooling and processes for adding, maintaining, and discovering documentation.
Design, build, deploy, and maintain Sift, an agentic data exploration and analysis tool that supports data access, collaboration on insights, and data-driven decision-making.