Software Engineer, Big Data, Tech Enablement
About The Position
The Tech Enablement team builds the scalable, reliable, and efficient infrastructure, tools, and processes that help ZipRecruiter’s engineering organization deliver products and services. The team develops foundational systems for data infrastructure, AI enablement, data processing, ingestion, discoverability, pipeline orchestration, documentation, analysis, machine-learning pipelines, and experimentation. In this role, you will design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and through event streams for downstream applications. You will work with large, evolving datasets using batch and streaming methods, making trade-offs among speed, efficiency, and reliability. The position also supports machine-learning development through MLOps frameworks, improves data documentation, and contributes to Sift, an agentic data exploration and analysis tool.
Requirements
- At least 2 years of professional software development experience, including building and operating data pipelines that feed a customer-facing product.
- Hands-on Apache Spark experience writing, tuning, and debugging batch and/or streaming jobs at scale.
- Experience deploying and operating containerized services and batch jobs.
- Solid computer science fundamentals, including data structures, algorithms, distributed systems, and testing.
- Professional software development experience building and operating data pipelines and infrastructure that feed a customer-facing product.
Nice To Haves
- At least 4 years of professional software development experience focused on data-intensive systems.
- Working knowledge of AWS, including S3, IAM, EC2, and MSK or equivalents, and experience instrumenting jobs with metrics, logs, and alerts.
- Experience with a data lake stack, including Parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake.
- Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.
- Experience with a workflow orchestrator such as Apache Airflow, including authoring DAGs, backfills, dependency management, and on-call support for scheduled pipelines.
- Experience with open table format internals, including schema evolution, partitioning, and compaction.
- Experience deploying services on Kubernetes with Docker.
- Experience building and maintaining a data platform for a user-facing product, including infrastructure and tools for streaming and batch ingestion, ETL pipelines, data permissions management, and data discovery.
- BS, MS, or PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.
Responsibilities
- Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and through event streams for downstream applications.
- Process large, evolving datasets using batch and streaming techniques, balancing speed, efficiency, and reliability.
- Design, build, deploy, and maintain frameworks and systems that accelerate machine-learning development and improve its velocity, quality, observability, and reliability.
- Develop frameworks and systems that improve the comprehensiveness and quality of data documentation, and make documentation easier for people and automated processes to add, maintain, and discover.
- Design, build, deploy, and maintain Sift, an agentic data exploration and analysis tool that democratizes data access and supports collaboration and data-driven decisions.
Benefits
- Competitive compensation
- Exceptional benefits package
- Flexible Vacation & Paid Time Off
- Employer-matched 401(k) plan
- Depending on the position offered, equity, bonuses, commission, or other forms of compensation may be provided.
- Medical, financial, and other benefits
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
