Sr. Data Engineer ( big data )
About The Position
We have the following urgent role with our DIRECT client. This role requires strong engineering skills, an analytical approach, and good programming skills. The candidate will provide business insights by leveraging internal tools and systems, databases, and industry data. The position involves coding, analytical modeling, root cause analysis, investigation, debugging, testing, and collaboration with business partners, product managers, and other engineering teams. The role requires a minimum of 5+ years of experience, with experience in the retail business being a plus. Excellent written and verbal communication skills are necessary for communicating engineering subject matter to varied audiences. The ability to document requirements, data lineage, and subject matter in both business and technical terminology is also essential. The candidate will guide and learn from other team members and demonstrate the ability to transform business requirements into code, specific analytical reports, and tools.
Requirements
- Development experience with Java, Scala, Flume, Python.
- Development background with Spark or MapReduce/YARN is a must-have.
- Significant application development experience in the past either with Java or Python.
- Experience as a Software Engineer who transitioned to a Data Engineer.
- Knowledge/experience on Teradata Physical Design and Implementation.
- Knowledge/experience on Teradata SQL Performance Optimization.
- Advanced SQL (preferably Teradata).
- Experience working with large data sets.
- Experience working with distributed computing (MapReduce, Hadoop, Hive, Pig, Apache Spark, etc.).
- Strong Hadoop scripting skills to process petabytes of data.
- Experience in Unix/Linux shell scripting or similar programming/scripting knowledge.
- Experience in ETL processes.
- Experience with real-time data ingestion (Kafka).
- Minimum of 5+ years’ experience.
- Excellent written and verbal communication skills for varied audiences on engineering subject matter.
- Ability to document requirements, data lineage, subject matter in both business and technical terminology.
Nice To Haves
- Experience with Teradata Tools and Utilities (FastLoad, MultiLoad, BTEQ, FastExport).
- Cassandra experience.
- Automic scheduler experience.
- R/R studio, SAS experience.
- Presto experience.
- Hbase experience.
- Tableau or similar reporting/dashboarding tool experience.
- Modeling and Data Science background.
- Retail industry background.
Responsibilities
- Build end-to-end data pipelines from the ground up.
- Develop and implement solutions using Java, Scala, Flume, and Python.
- Develop applications using Spark or MapReduce/YARN.
- Perform Teradata Physical Design and Implementation.
- Optimize Teradata SQL performance.
- Work with large data sets and distributed computing environments (MapReduce, Hadoop, Hive, Pig, Apache Spark, etc.).
- Process petabytes of data using strong Hadoop scripting skills.
- Write Unix/Linux shell scripts or similar programming/scripting code.
- Implement ETL processes.
- Handle real-time data ingestion using Kafka.
- Transform business requirements into code, analytical reports, and tools.
- Conduct root cause analysis, investigation, debugging, and testing.
- Collaborate with business partners, product managers, and other engineering teams.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
