Senior Software Engineer, Big Data, Tech Enablement

ZipRecruiter•Santa Monica, CA

1d•Hybrid

About The Position

The Senior Software Engineer will join ZipRecruiter’s Tech Enablement team, which builds scalable, reliable, and efficient infrastructure, tools, and processes that help engineering teams deliver products and services. The team provides foundational systems for data infrastructure and AI enablement, helping engineers and data scientists focus on products rather than underlying technologies. The role focuses on designing, building, deploying, and maintaining data ingestion and datalake systems that make web events available in the datalake and through event streams for downstream applications. It also involves processing large, evolving datasets using batch and streaming approaches, and developing MLOps frameworks that improve the velocity, quality, observability, and reliability of machine learning development and ML-powered products. Additional areas of work include building frameworks and tooling to improve data documentation, and developing Sift, an agentic data exploration and analysis tool intended to democratize access to data, support collaboration on insights, and accelerate data-driven decision-making. The position offers a hybrid work environment; most US-based positions may also be performed remotely, subject to exceptions noted in the minimum qualifications.

Requirements

At least 4 years of professional software development experience, including building and operating data pipelines that feed into a customer-facing product.
Hands-on Apache Spark experience writing, tuning, and debugging batch and/or streaming jobs at scale.
Experience deploying and operating containerized services and batch jobs.
Solid computer science fundamentals, including data structures, algorithms, distributed systems, and testing.
Professional software development experience building and operating data pipelines and infrastructure that feed into a customer-facing product.

Nice To Haves

At least 4 years of professional software development experience focused on data-intensive systems.
Working knowledge of AWS, including S3, IAM, EC2, and MSK or equivalents, and experience instrumenting jobs with metrics, logs, and alerts.
Experience with a data lake stack, including Parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake.
Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.
Experience with a workflow orchestrator such as Apache Airflow, including authoring DAGs, backfills, dependency management, and on-call support for scheduled pipelines.
Experience with open table format internals, including schema evolution, partitioning, and compaction.
Experience deploying services on Kubernetes with Docker.
Experience building and maintaining a data platform powering a user-facing product, including infrastructure and tools for streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery.
BS, MS, or PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.

Responsibilities

Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and via event streams for downstream applications.
Process large, evolving datasets using batch and streaming techniques, balancing speed, efficiency, and reliability.
Design, build, deploy, and maintain MLOps frameworks and systems that accelerate machine learning development and improve its velocity, quality, observability, and reliability.
Design, build, deploy, and maintain frameworks and systems that improve the comprehensiveness and quality of data documentation, including tooling and processes for adding, maintaining, and discovering documentation.
Design, build, deploy, and maintain Sift, an agentic data exploration and analysis tool.
Build and operate data pipelines and infrastructure that feed into customer-facing products.