Scala Sparks Developer

TekWissen • New York City, NY

7d • Hybrid

About The Position

TekWissen is seeking a Scala Sparks Developer for a six-month temporary assignment in New York City, NY. The role is with a client described as a global professional services firm specializing in digital transformation, business process management, analytics, and AI-driven solutions.

The developer will build and maintain scalable data-processing pipelines with Scala and Apache Spark, create data transformation and ETL workflows for structured and unstructured data, and use Python for processing, scripting, and Spark workflow integration. The position also involves optimizing Spark applications, contributing to code reviews and design discussions, following best practices, and documenting workflows, architecture, and solutions for internal knowledge sharing.

Requirements

• Strong experience with Scala and Apache Spark, including Spark Core, Spark SQL, DataFrames, and RDDs.
• Proficiency in Python for data manipulation, scripting, and automation.
• Proven experience with distributed computing concepts and large-scale data processing.
• Knowledge of ETL development, data warehousing, and data modeling techniques.
• Excellent communication and collaboration skills.

Nice To Haves

• Familiarity with Big Data ecosystems such as Hadoop, Hive, Kafka, or HBase.
• Experience with cloud platforms such as AWS, GCP, or Azure, particularly with Spark services.
• Familiarity with streaming data processing using Spark Streaming or Structured Streaming.
• Knowledge of data governance, security, and compliance practices.
• Basic understanding of machine learning workflows.

Responsibilities

• Develop, test, and maintain scalable data processing pipelines using Scala and Apache Spark.
• Implement data transformation and ETL workflows for structured and unstructured data.
• Use Python for data processing, scripting, and integrating Spark workflows.
• Optimize Spark application performance through tuning, partitioning, and caching strategies.
• Participate in code reviews and design discussions and ensure adherence to best practices.
• Document workflows, architecture, and solutions for internal knowledge sharing.

What This Job Offers

• Education Level: No Education Listed