Senior Hadoop Developer

The Timberline Group•Saint Louis, MO

1d•Remote

About The Position

We are seeking a Senior Hadoop Developer to develop, create, and modify general computer applications software or specialized utility programs. This role involves implementing data analytics processing algorithms on Big Data frameworks, performing data acquisition and analysis, and managing complex data-related issues. The developer will be responsible for loading data, sourcing large volumes of data into the Hadoop platform, and installing, configuring, and maintaining the enterprise Hadoop environment. Additionally, the role includes building scalable data pipelines, defining and managing Hadoop jobs, reviewing logs, designing schemas for Hive and HBase, and mentoring other developers. The position also requires developing efficient Pig and Hive scripts, optimizing HDFS formats, troubleshooting Hadoop ecosystem issues, and documenting technical designs. The developer will also design and develop data integration solutions, lead technical meetings, implement Spark Streaming, develop API services in Java Spring, and create Hive/HBase connections to Spring. Multi-threading in Java/Scala is also a key responsibility.

Requirements

Bachelor’s degree in Computer Science, Applied Computer Science, Engineering, or any related field of study.
At least two (2) years of experience in the job offered or in any related position(s).
Demonstrable proficiency, skill, experience, and knowledge with Hadoop/Big Data Ecosystem and Architecture.
Demonstrable proficiency, skill, experience, and knowledge with Hive, Spark, HBase, Sqoop, Impala, Kafka, Flume, Oozie, and MapReduce.
Demonstrable proficiency, skill, experience, and knowledge with programming in Java, Scala, Python, and Shell Scripting.
Demonstrable proficiency, skill, experience, and knowledge with SQL and Data modelling.

Responsibilities

Implement data analytics processing algorithms on Big Data batch and stream processing frameworks (e.g. Hadoop MapReduce, Python, Spark, Scala, Kafka etc.).
Perform data acquisition, preparation, and analysis leveraging a variety of data programming techniques in Spark using Scala.
Work on complex issues where analysis of situations and data requires an in-depth evaluation of variable factors.
Load data from different datasets and decide on which file format is efficient for a task.
Source large volumes of data from diverse data platforms into Hadoop platform.
Install, configure, and maintain enterprise Hadoop environment.
Build distributed, reliable, and scalable data pipelines to ingest and process data in real-time.
Deal with fetching impression streams, transaction behaviors, clickstream data, and other unstructured data.
Define Hadoop Job Flows and manage Hadoop jobs using Scheduler.
Review and manage Hadoop log files.
Design and implement column family schemas of Hive and HBase within HDFS.
Assign schemas and create Hive tables with suitable formats and compression techniques.
Mentor Big Data Developers on best practices and strategic development.
Develop efficient Pig and Hive scripts with joins on datasets using various techniques.
Apply different HDFS formats and structure like Parquet, Avro, etc. to speed up analytics.
Fine tune Hadoop applications for high performance and throughput.
Troubleshoot and debug any Hadoop ecosystem run time issues.
Develop and document technical design specifications.
Design and develop data integration solutions (batch and real-time) to support enterprise data platforms including Hadoop, RDBMS, and NoSQL.
Lead technical meetings, as required, and convey ideas clearly and tailor communication based on selected audience (technical and non-technical).
Implement Spark Streaming architecture and integration with JMS queue with custom receivers.
Develop and deploy API services in Java Spring.
Create Hive and HBase data source connection to Spring.
Implement multi-threading in Java/Scala.