Lead Software Engineer - Big Data/Spark/Hadoop/Databricks

JPMorgan Chase - Jersey City, NJ

Full-time

Jersey City, NJ

Credit Intermediation and Related Activities

About the position

As a Lead Software Engineer at JPMorgan Chase, you will play a pivotal role in the Corporate Technology Engineering and Architecture team. This position involves designing and delivering advanced data and technology products while ensuring they are secure, stable, and scalable. You will lead initiatives related to data cataloging, contributing to the firm's data governance strategy, and will be responsible for overseeing critical technology solutions across various business functions.

Responsibilities

• Oversee the coordination of activities concerning metadata identification, standardization, and updates within data lakehouse tables.
• Develop and publish metrics for metadata completeness and data catalog usage to foster wider adoption.
• Serve as a subject matter expert for data catalog, creating training materials and educating users as needed.
• Execute software solutions, design, development, and technical troubleshooting, thinking beyond conventional approaches to build solutions.
• Create secure and high-quality production code and maintain algorithms that run synchronously with appropriate systems.
• Produce architecture and design artifacts for complex applications, ensuring design constraints are met by software code development.
• Gather, analyze, synthesize, and develop visualizations and reporting from large, diverse data sets for continuous improvement of software applications and systems.
• Proactively identify hidden problems and patterns in data to drive improvements to coding hygiene and system architecture.
• Contribute to software engineering communities of practice and events that explore new and emerging technologies.
• Add to team culture of diversity, equity, inclusion, and respect.

Requirements

• 10 years of experience in software engineering, including 5 years of hands-on expertise in ETL/Data pipeline and data lake platforms like Databricks, Spark/Hadoop, and Snowflake.
• Strong understanding of data management principles, governance frameworks, and best practices.
• Experience with data cataloging tools and systems (e.g., Collibra, Alation, Informatica, Atlan, Databricks Unity Catalog).
• Experience in developing, debugging, and maintaining code in a large enterprise with one or more modern programming languages (e.g., Java/SpringBoot, Python/Flask).
• Expert in SQL and data profiling tools.
• Solid understanding of SDLC practices, Application Resiliency, Security, Agile methodologies, and CI/CD.
• Knowledge of integration technologies (e.g., GraphQL, REST).
• Excellent analytical and problem-solving skills.
• Strong attention to detail and organizational skills.
• Effective communication and interpersonal skills.
• Ability to work independently as well as in a team environment.

Nice-to-haves

• Practical cloud native experience, with exposure to designing and deploying applications on AWS.
• Hands-on experience with data reporting and BI Tools like Tableau, Alteryx.