Senior Big Data Engineer
About The Position
Performance Optimization Analyze and optimize existing Hadoop/Spark pipelines to improve processing speed, resource utilization, and reliability Identify bottlenecks in data workflows and implement solutions that reduce processing time and costs Tune Spark jobs, Hive queries, and Impala performance through partitioning strategies, caching, and execution plan optimization Design and build scalable data pipelines using Spark (Scala) to process terabytes of data efficiently Develop robust ETL/ELT workflows that integrate data from multiple sources into Hadoop environment and Oracle data warehouses Implement data quality checks and monitoring to ensure pipeline reliability Work closely with product teams to understand requirements and deliver data solutions Participate in code reviews and contribute to engineering best practices
Requirements
- 6+ years of hands-on experience with Hadoop ecosystem technologies
- Experience in managing and implementing successful projects
- Strong proficiency in Apache Spark with Scala development
- Solid experience with Hive and Impala for large-scale data querying
- Understanding of distributed computing principles and data partitioning strategies
- Experience optimizing Spark jobs and SQL queries for performance
- Proficiency with version control (Git) and CI/CD practices
- Proficiency with streaming frameworks (Kafka)
Nice To Haves
- Exposure to cloud platforms and technologies is preferred
- Exposure to Databricks is preferred
- Master's degree preferred
Responsibilities
- Analyze and optimize existing Hadoop/Spark pipelines
- Identify bottlenecks in data workflows
- Tune Spark jobs, Hive queries, and Impala performance
- Design and build scalable data pipelines using Spark (Scala)
- Develop robust ETL/ELT workflows
- Implement data quality checks and monitoring
- Work closely with product teams to understand requirements and deliver data solutions
- Participate in code reviews and contribute to engineering best practices
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Number of Employees
5,001-10,000 employees
