Engineer
About The Position
Senior Data Engineer with 6 to 8 Years of strong expertise in Python programming and Big Data technologies (Hadoop ecosystem) to design, develop, and optimize large-scale data pipelines. Design and implement scalable ETL pipelines using Python and Hadoop tools (Hive, Pig, HDFS). Automate data ingestion and transformation for structured and unstructured data. Develop distributed data processing workflows using Spark, MapReduce, and Hadoop. Optimize performance for high-volume datasets. Create efficient data models for analytics and reporting. Implement best practices for data partitioning and indexing in Hadoop. Monitor and tune Hadoop clusters for optimal performance. Ensure high availability and fault tolerance of data systems. Work closely with data scientists and analysts to enable predictive analytics and machine learning.
Requirements
- 6 to 8 Years of strong expertise in Python programming and Big Data technologies (Hadoop ecosystem)
- Strong Python programming for data processing and automation
- Expertise in Hadoop ecosystem (HDFS, Hive, Pig, MapReduce)
- Experience with Spark for distributed computing
- Proficiency in SQL and NoSQL databases
- Strong analytical and problem-solving skills
- Experience with Agile/Scrum methodologies
Responsibilities
- Design and implement scalable ETL pipelines using Python and Hadoop tools (Hive, Pig, HDFS)
- Automate data ingestion and transformation for structured and unstructured data
- Develop distributed data processing workflows using Spark, MapReduce, and Hadoop
- Optimize performance for high-volume datasets
- Create efficient data models for analytics and reporting
- Implement best practices for data partitioning and indexing in Hadoop
- Monitor and tune Hadoop clusters for optimal performance
- Ensure high availability and fault tolerance of data systems
- Work closely with data scientists and analysts to enable predictive analytics and machine learning
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Industry
Professional, Scientific, and Technical Services
Education Level
No Education Listed
Number of Employees
5,001-10,000 employees
