Lead Data Engineer

Innova Solutions•Phoenix, AZ

15h•Onsite

About The Position

A client of Innova Solutions is immediately hiring for a Lead Data Engineer position. This is a full-time, 12-month contract role with opportunities to work in Charlotte, NC, Irving, TX, Phoenix, AZ, Columbus, OH, or Minneapolis, MN. As a Data Engineer, you will be responsible for creating and supporting Spark-based processing and Kafka streaming, developing data flows using Kafka, Flink, and Spark streaming, and designing and implementing highly scalable and modular data pipelines with built-in data controls for automating data governance.

Requirements

10+ years of experience in data engineering including hands-on experience working with Hadoop and Google Cloud data solutions
Hands-on experience developing data flows using Kafka, Flink, and Spark streaming
Experience with Data Lakehouse architecture and design
Experience with Python, pySpark, Apache Kafka, Airflow, and SQL, GPC Cloud Storage, BigQuery, Data Proc, Cloud Composer
Experience with NoSQL databases such as columnar databases, graph databases, document databases, KV stores, and associated data formats

Nice To Haves

Proven skills with data migration from on-prem to a cloud native environment
Proven experience working with the Hadoop ecosystem capabilities such as Hive, HDFS, Parquet, Iceberg, and Delta Tables
Design and implementation of highly scalable and modular data pipelines with built-in data controls for automating data governance
Familiarity of GenAI frameworks such as Langchain and Langraph to develop agent-based data capabilities
Dev Ops and CI/CD deployments including Git, Jenkins, Docker, and Kubernetes
Web based UI development using React and Node JS is a plus

Responsibilities

Creating/supporting Spark based processing, Kafka streaming
Developing data flows using Kafka, Flink, and Spark streaming
Designing and implementing of highly scalable and modular data pipelines with built-in data controls for automating data governance