Big Data Lead
About The Position
The Big Data Lead will be responsible for implementing and managing big data processing workflows, including ETL pipelines and data transformations. This role involves ensuring data quality and integrity, troubleshooting PySpark applications, and integrating PySpark code with various frameworks. The lead will also be responsible for documenting code lineage and ensuring compliance with data security and privacy regulations. The position requires strong programming skills in Python and SQL, extensive experience with big data technologies like Spark, Hadoop, Hive, and Kafka, and an understanding of data warehousing concepts and DevOps practices. Excellent problem-solving, analytical, communication, and leadership abilities are essential.
Requirements
- Experience with big data processing and distributed computing systems like Spark
- Strong programming skills in Python and SQL
- Experience with big data technologies like Hadoop, Hive, and Kafka
- Understanding of data warehousing concepts and relational databases like SQL
- Knowledge of CI/CD pipelines and DevOps practices
- Strong problem-solving and analytical skills
- Excellent communication and leadership abilities
- 4+ years of experience in big data development, Hadoop, Hive & Spark framework
- Strong Python, PySpark Development and SQL knowledge
Nice To Haves
- Experience in SAS
- Certification in big data or cloud technologies
Responsibilities
- Implement ETL pipelines and data transformation processes
- Ensure data quality and integrity in all data processing workflows
- Troubleshoot and resolve issues related to PySpark applications and workflows
- Understand source, dependencies and data flow from converted PySpark code
- Demonstrate and document code lineage
- Integrate PySpark code with frameworks such as Ingestion Framework, DataLens, etc.
- Ensure compliance with data security, privacy regulations, and organizational standards
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
Number of Employees
5,001-10,000 employees
