About The Position
We are hiring an SRE to own reliability, performance, data freshness, and operational readiness for our production ETL platform. The platform supports data ingestion, transformation, and loading workflows across Kubernetes-based environments — including Airflow-based loader jobs and Spark-on-EKS jobs that load data into a Datalake/Lakehouse. You will operate and triage production pipelines end-to-end: extractors, loaders, batch jobs, streaming ingestion (Kafka), Spark workloads, and Airflow DAGs. You will tune Kubernetes and Spark for stability, build observability tooling, drive root cause analysis, write automation to reduce toil, and manage configuration and secrets through GitOps-style processes. The ideal candidate troubleshoots distributed systems from logs, metrics, and infrastructure signals, and drives permanent fixes through engineering partnership.
Requirements
- 4+ years of experience in SRE, DevOps, platform engineering, data infrastructure, or production operations.
- Strong Linux troubleshooting skills.
- Strong Kubernetes/EKS operations experience (kubectl, deployments, pods, resource limits, service accounts, secrets, workload debugging).
- Hands-on experience supporting Apache Spark on Kubernetes — including tuning, log analysis, memory issues, shuffle failures, and performance bottlenecks.
- Production experience with Apache Airflow (DAG operations, task failures, retries, scheduling, SLA misses).
- Production experience with Kafka or similar streaming platforms (consumer groups, lag, offsets, partitions, secure client connectivity).
- Familiarity with Datalake/Lakehouse architectures.
- Familiarity with object storage (S3 or equivalent).
- Familiarity with monitoring/logging tools (Splunk, Prometheus, Grafana, CloudWatch, ELK, Datadog).
- Solid SQL skills.
- Scripting proficiency in Python and Bash.
- Strong incident management, RCA, change management, and operational documentation skills.
Nice To Haves
- Experience supporting metadata-driven ETL platforms or internal ETL frameworks.
- Experience with Lakehouse technologies (Spark, Iceberg, IRC, Hive Metastore, Parquet) and data loading performance tuning.
- Experience with GitOps or source-of-truth configuration management, and infrastructure-as-code tools (Terraform, Helm, Argo CD, Ansible).
- Experience operating multi-region or region-specific data platforms.
- Experience with certificate/PKI/TLS management, JKS/truststore, Kerberos, or key rotation processes.
- Familiarity with Spark performance tuning at scale.
- Familiarity with platform dependencies such as API gateways, RabbitMQ, Redis, or Cassandra.
Responsibilities
- Own reliability, performance, data freshness, and operational readiness for the production ETL platform.
- Operate and triage production pipelines end-to-end, including extractors, loaders, batch jobs, streaming ingestion (Kafka), Spark workloads, and Airflow DAGs.
- Tune Kubernetes and Spark for stability.
- Build observability tooling.
- Drive root cause analysis.
- Write automation to reduce toil.
- Manage configuration and secrets through GitOps-style processes.
- Troubleshoot distributed systems from logs, metrics, and infrastructure signals.
- Drive permanent fixes through engineering partnership.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
