Senior Data Engineer

CGI•Strongsville, OH

48d•Onsite

About The Position

We are seeking a highly motivated and experienced Senior Data Engineer to design, develop, and optimize large scale data transformation solutions on modern Big Data platforms. The ideal candidate is passionate about technology, thrives in a fast paced environment, and possesses a strong ownership mindset with a "can do" attitude. This role requires deep expertise in Hadoop ecosystem technologies, Python, and modern data lake architectures. The successful candidate will serve as a technical leader on a large scale digital transformation initiative, collaborating with cross functional business and technology teams while guiding both onshore and offshore development teams to deliver high quality, scalable data solutions. Experience with Generative AI technologies and AI assisted development is highly desirable. This position can be performed onsite five days a week at our client site in Strongsville, OH or Pittsburgh, PA or Dallas, TX. Future duties and responsibilities . Design, develop, and maintain scalable data pipelines and transformation frameworks using Hadoop ecosystem technologies. . Build high performance data transformation solutions utilizing Hive, Spark, Python, Impala, and Apache Iceberg. . Develop robust ETL/ELT processes supporting enterprise scale analytics and data products. . Design and optimize Hive, Spark SQL, and Impala queries for large datasets. . Implement scalable data lake solutions leveraging Apache Iceberg table formats and best practices. . Collaborate closely with Product Owners, Business Analysts, Subject Matter Experts, Architects, and Technical Managers to translate business requirements into technical solutions. . Lead technical design discussions and establish engineering best practices across the development team. . Provide technical leadership and mentoring to both onshore and offshore development teams. . Conduct code reviews and ensure adherence to coding standards, performance optimization, and data quality best practices. . Troubleshoot production issues and implement long term sustainable solutions. . Drive continuous improvement through automation, reusable frameworks, and engineering best practices. . Participate in Agile ceremonies including sprint planning, backlog refinement, and retrospectives. . Stay current with emerging Big Data, AWS cloud, and AI technologies and recommend innovative solutions.

Requirements

6+ years of experience in Data Engineering, Big Data development, or enterprise data platforms.
Strong experience developing enterprise scale Big Data applications.
Hands on expertise with: Hadoop ecosystem, Hive, Apache Spark, Impala, PySpark, Apache Iceberg.
Strong SQL development and query optimization skills.
Experience designing scalable ETL/ELT pipelines.
Experience working with distributed computing environments and large volume datasets.
Strong analytical, troubleshooting, and problem solving skills.
Experience working in Agile/Scrum delivery environments.
Excellent communication and stakeholder collaboration skills.
Proven ability to lead technical initiatives across geographically distributed teams.
Experience with cloud based data platforms (AWS, Azure, or Google Cloud).
Experience with orchestration tools such as Airflow or Oozie.
Experience with version control systems such as Git and CI/CD pipelines.
Familiarity with data governance, metadata management, and data quality frameworks.
Experience with performance tuning and optimization of Big Data workloads.

Nice To Haves

Experience with Generative AI technologies and AI assisted development is highly desirable.
Working knowledge of Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) architectures is a plus.

Responsibilities

Design, develop, and maintain scalable data pipelines and transformation frameworks using Hadoop ecosystem technologies.
Build high performance data transformation solutions utilizing Hive, Spark, Python, Impala, and Apache Iceberg.
Develop robust ETL/ELT processes supporting enterprise scale analytics and data products.
Design and optimize Hive, Spark SQL, and Impala queries for large datasets.
Implement scalable data lake solutions leveraging Apache Iceberg table formats and best practices.
Collaborate closely with Product Owners, Business Analysts, Subject Matter Experts, Architects, and Technical Managers to translate business requirements into technical solutions.
Lead technical design discussions and establish engineering best practices across the development team.
Provide technical leadership and mentoring to both onshore and offshore development teams.
Conduct code reviews and ensure adherence to coding standards, performance optimization, and data quality best practices.
Troubleshoot production issues and implement long term sustainable solutions.
Drive continuous improvement through automation, reusable frameworks, and engineering best practices.
Participate in Agile ceremonies including sprint planning, backlog refinement, and retrospectives.
Stay current with emerging Big Data, AWS cloud, and AI technologies and recommend innovative solutions.