Data Engineer – Big Data (Spark/Scala, Hadoop, SQL, Python)

Computer Task Group, Inc•Piscataway Township, NJ

4d•Onsite

About The Position

CTG is seeking to fill a Data Engineer – Big Data (Spark/Scala, Hadoop, SQL, Python) position for our client. This is a 12-month contract role located in Piscataway, NJ. The position involves designing, developing, testing, and maintaining scalable data-processing applications and Big Data pipelines. The role requires strong analytical and problem-solving skills, with a focus on data quality, validation, and reconciliation. Collaboration within an Agile environment and contribution to API development are also key aspects of this role.

Requirements

Strong hands-on experience with Apache Spark and Scala for distributed data processing.
Strong experience with Hadoop, Hive, and Impala and enterprise Big Data platforms.
Advanced SQL skills, including complex queries, joins, transformations, reconciliation, and performance tuning.
Strong Python scripting and data-processing experience.
Strong application development, coding, debugging, and problem-solving skills.
Knowledge of ETL/data pipelines, data quality, and data reconciliation.
Understanding of analytics libraries, statistical computing, and open-source data-processing technologies.
Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent combination of education and experience.
Excellent verbal and written English communication skills and the ability to interact professionally with a diverse group.

Nice To Haves

Experience with Agile development methodologies and CI/CD practices is preferred.
Experience with large-volume transactional or financial datasets is preferred.
Banking or financial services experience is preferred.
Experience supporting production data applications and resolving technical issues is a plus.

Responsibilities

Design, develop, test, and maintain scalable data-processing applications using Apache Spark and Scala.
Develop and support Big Data applications and data pipelines utilizing Hadoop, Hive, and Impala.
Write complex SQL queries for data extraction, transformation, reconciliation, analysis, and performance optimization.
Develop Python scripts and utilities to support data processing, automation, and data engineering activities.
Design, build, and maintain scalable ETL/data pipelines for large-volume datasets.
Analyze, troubleshoot, and debug complex application and data-processing code.
Support data quality, validation, reconciliation, and integrity across enterprise data platforms.
Collaborate with application developers, data engineers, analysts, and business stakeholders in an Agile environment.
Contribute to API development and application solutions supporting Big Data platforms and analytics.
Apply knowledge of analytics libraries, open-source technologies, statistical computing, and Big Data processing frameworks as appropriate.