VP II AI, Machine Learning|US Remote

Circana•,

1d•$202,000 - $210,000•Remote

About The Position

The Senior Big Data Engineer is responsible for designing, building, and delivering highly scalable big data and ETL solutions across distributed environments. This role supports projects at various and unanticipated client worksites throughout the United States, with headquarters based in Chicago, IL. The engineer works closely with cross-functional teams to develop, test, and deploy data solutions that meet business and operational needs, leveraging modern big data frameworks, cloud platforms, and agile development practices. Telecommuting is permitted.

Requirements

Design and implement highly scalable ETL applications using Hadoop and Big Data ecosystems.
Develop new scripts, tools, and methodologies to streamline and automate ETL workflows.
Deliver big data solutions using Spark, Python, Scala, SQL, and Hive.
Design and create use cases and scenarios for functional, integration, and system testing.
Collaborate closely with Data Science, QA, Operations, and other teams to meet aggressive deadlines.
Participate in daily Agile/Scrum meetings and conduct code reviews.
Coordinate with cross-functional operational teams to manage end-to-end data delivery.
Write efficient, reusable, and well-documented code.
Prepare technical design documentation for implemented solutions.
Identify and resolve data pipeline issues and provide timely solutions for incorrect or undesired system behavior using ILD solutions.
Work within complex technical environments that include: ETL Spark applications in PySpark and Scala, Spark architecture, data frames, and performance tuning, Hadoop ecosystem tools such as Flume, MapReduce, and HDFS, Relational databases (Oracle, PostgreSQL), Big Data querying tools (Hive, Pig, Impala), Databricks, distributed computing principles, and Lambda Architecture, Cloud platforms such as Microsoft Azure.

Nice To Haves

Stay Curious: Being hungry to learn and grow, always asking the big questions.
Seek Clarity: Embracing complexity to create clarity and inspire action.
Own the Outcome: Being accountable for decisions and taking ownership of our choices.
Center on the Client: Relentlessly adding value for our customers.
Be a Challenger: Never complacent, always striving for continuous improvement.
Champion Inclusivity: Fostering trust in relationships engaging with empathy, respect, and integrity.
Commit to each other: Contributing to making Circana a great place to work for everyone.

Responsibilities

Design and implement highly scalable ETL applications using Hadoop and Big Data ecosystems.
Develop new scripts, tools, and methodologies to streamline and automate ETL workflows.
Deliver big data solutions using Spark, Python, Scala, SQL, and Hive.
Design and create use cases and scenarios for functional, integration, and system testing.
Collaborate closely with Data Science, QA, Operations, and other teams to meet aggressive deadlines.
Participate in daily Agile/Scrum meetings and conduct code reviews.
Coordinate with cross-functional operational teams to manage end-to-end data delivery.
Write efficient, reusable, and well-documented code.
Prepare technical design documentation for implemented solutions.
Identify and resolve data pipeline issues and provide timely solutions for incorrect or undesired system behavior using ILD solutions.
Work within complex technical environments that include: ETL Spark applications in PySpark and Scala, Spark architecture, data frames, and performance tuning, Hadoop ecosystem tools such as Flume, MapReduce, and HDFS, Relational databases (Oracle, PostgreSQL), Big Data querying tools (Hive, Pig, Impala), Databricks, distributed computing principles, and Lambda Architecture, Cloud platforms such as Microsoft Azure.