Senior ETL Developer - Assistant Vice President

Citi•Tampa, FL

31d•$96,960 - $145,440•Onsite

About The Position

Senior ETL Developer will be responsible for designing, implementing, and optimizing distributed data processing jobs to handle large-scale data in Hadoop Distributed File System(HDFS) using Apache Spark and Python. This role required deep understanding of data engineering principles, proficiency in Python and hands-on experience with Spark and Hadoop ecosystems. Developer will collaborate with data engineers, analysts, and business stakeholders to process, transform and drive insights and data driven decisions.

Requirements

5-8 years of relevant experience in Software Development
Experience in systems analysis and programming of software applications
Experience in managing and implementing successful projects
Working knowledge of consulting/project management techniques/methods
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Proficiency in Python
Hands-on experience with Spark and Hadoop ecosystems
Deep understanding of data engineering principles

Responsibilities

Design and Implement of Spark applications to process and transform large datasets in HDFS.
Develop ETL Pipelines in Spark using Python for data Ingestion, cleaning, aggregation, and transformations.
Optimize Spark jobs for efficiency, reducing run time and resource usage.
Finetune memory management, caching, and partitioning strategies for Optimal performance.
Load data from different sources into HDFS, ensuring data accuracy and integrity.
Integrate Spark Applications with Hadoop frameworks like Hive, Sqoop etc.
Troubleshoot and debug Spark Job failures, monitor job logs, and Spark UI to Identify Issues.