Scala - Spark Developer

Morgan Stanley • New York, NY

1d • $90,000 - $150,000

About The Position

We are looking for a Scala - Spark Developer to join our regulatory reporting data engineering team. The application processes very large datasets, with business logic implemented using Scala/Spark workloads on Databricks and Snowflake as the core data platform. The selected candidate will work closely with senior engineers to develop data pipelines, implement controls, and support testing and operational activities.

Requirements

• Bachelor’s degree in Computer Science, Engineering, MIS, or a related field
• 3+ years of experience in software development or data engineering
• Strong foundational knowledge of Scala and familiarity with Apache Spark
• Understanding of cloud data platforms (Snowflake preferred)
• Database knowledge and core database concepts(SQL, indexing, joins, transaction, query optimization)
• Ability to understand and debug spark application(DAGS, Job, Stages, Failure)
• Understanding of spark job optimization techniques
• Working knowledge of Python for scripting, automation, and testing
• Exposure to BDD frameworks and writing unit tests
• Strong understanding of SDLC, coding practices, and Agile methodology
• Good communication skills and willingness to learn complex financial datasets

Nice To Haves

• Basic understanding of Trade Lifecycle concepts
• Awareness of Equities and Options asset classes
• Exposure to Front Office (FO) trade/order data
• Experience with Unix/Linux commands and shell scripting
• Familiarity with batch job schedulers (Autosys is a plus)
• Basic exposure to Generative AI use cases
• Experience with large-scale data environments

Responsibilities

• Assist in developing and enhancing Spark-based dat pipelines written in Scala on Databricks
• Implement and maintain business logic for regulatory reporting under guidance of senior team members
• Develop automation utilities and operational controls using Python
• Work with datasets stored in Snowflake including data transformations and validations
• Support batch workflow setup and maintenance using Autosys scheduler
• Write and maintain BDD test cases and Python unit tests
• Participate in code reviews, design discussions, and Agile ceremonies
• Provide basic support during production issues and assist in root-cause analysis
• Ensure good documentation, coding standards, and quality practices

Benefits

• Hands-on learning with modern tech stack: Scala, Spark, Databricks, Snowflake
• Opportunity to work on large-scale, high-impact regulatory reporting systems
• Strong mentoring and skill-building environment with senior engineers
• Collaborative & respectful culture focused on growth, learning, and continuous improvement

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level