Sr. Big Data Engineers| HQ: Chicago, IL

Circana•,

4d•$160,500 - $200,000•Remote

About The Position

The Senior Big Data Engineer is responsible for designing, building, and delivering highly scalable big data and ETL solutions across distributed environments. This role supports projects at various and unanticipated client worksites throughout the United States, with headquarters based in Chicago, IL. The engineer works closely with cross-functional teams to develop, test, and deploy data solutions that meet business and operational needs, leveraging modern big data frameworks, cloud platforms, and agile development practices. Telecommuting is permitted.

Requirements

Bachelor’s degree in Computer Science or Information Systems or Computer Engineering or any Engineering or related field plus 5 years of progressive experience as a Software Engineer/Developer or in Software Development required.
experience writing ETL Spark applications in PySpark or Scala
spark architecture, data frames, tuning spark
relational databases (Oracle, PostgreSQL)
Python, SQL HQL, Hive
Data Bricks
managing software systems using Hadoop, MapReduce, HDFS and all included services
experience with distributed computing principles
Big Data querying tools (Hive)
data-warehousing & data-modeling techniques
Core Java, Linux, SQL, scripting languages
Cloud platforms (Azure)
integration of data from multiple data sources
Lambda Architecture
Telecommuting permitted.
4 years of university study in Computer Science or Information Systems or Computer Engineer plus 7 years of progressive experience as a Software Engineer/Developer or in software development in lieu of Bachelor's degree plus 5 years of progressive experience as a software Engineer/Developer or in software development.

Nice To Haves

Flume
spark architecture, data frames, tuning spark
relational databases (Oracle, PostgreSQL)
Python, SQL HQL, Hive
Data Bricks
managing software systems using Hadoop, MapReduce, HDFS and all included services
distributed computing principles
Big Data querying tools (Pig, Hive, Impala)
data-warehousing and data-modeling techniques
Core Java, Linux, SQL, scripting languages
Cloud platforms (Azure)
integration of data from multiple data sources
Lambda Architecture

Responsibilities

Design and implement highly scalable ETL applications on Hadoop and Big Data eco systems.
Develop new scripts, tools, and methodologies for streamlining and automating ETL workflows.
Deliver big data projects using Spark, Python, Scala, SQL, and Hive.
Design and create use-cases and scenarios for functional testing, integration testing, and system testing.
Work closely with Data Science, QA, Operations, and other teams to deliver on tight deadlines.
Participate in daily agile and scrum meetings and code reviews.
Coordinate with cross-functional operational teams for managing data delivery.
Write efficient, reusable, and well documented code.
Prepare technical design documents for solutions.
Identify and address issues encountered from the data factory and provide timely solutions to incorrect or undesired results or behavior with ILD solutions.