Sr. Databricks Data Engineer, Vice President

State Street•Princeton, NJ

10d•$120,000 - $202,500•Hybrid

About The Position

The Sr. Databricks Data Engineer will contribute hands-on to the development, enhancement, and maintenance of platforms as part of an agile scrum team. The role requires technical, people, and process skills, a strong background in application architecture and development, and a thorough understanding of leading IT technologies. The successful candidate will help deliver new products in the financial space using cutting-edge technology. The engineer designs, builds, and tests end-to-end data pipelines on cloud infrastructure, develops ETL processes, and works extensively with Databricks and data warehousing. The role encompasses the software development lifecycle, from design and coding through integration testing, deployment, and documentation. The position is onsite four days per week and remote one day per week, with standard business hours aligned to project and stakeholder needs (8 a.m. to 5 p.m.).

Requirements

Bachelor’s degree in a computer- or IT-related subject.
10+ years of overall Big Data data pipeline experience.
5+ years of hands-on Databricks experience.
5+ years of cloud-based development experience, including Azure services, Azure DevOps, Kubernetes, and Docker.
Hands-on development experience with Databricks SQL and Scala.
Proficient working knowledge of data warehousing platforms and technologies such as Databricks, Delta Lake, and Apache Spark.
Strong understanding of the Databricks platform, including clusters, jobs, and other resources.
Experience troubleshooting DevOps pipelines, including ADO and Harness.
Experience working in a multi-developer environment using version control such as Git.
Experience handling semi-structured data, including Avro, JSON, and XML.
Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer.
Strong critical-thinking, communication, and problem-solving skills.
Ability to communicate effectively and professionally, both orally and in writing.
Team-oriented, positive, enthusiastic, proactive, self-motivated, energetic, adaptable, and a fast learner.
Experience working in the financial industry.
Experience with agile development methodology.

Nice To Haves

Experience in microservices architecture and understanding of cloud computing; Azure is preferred.
Experience with Snowflake.

Responsibilities

Design, build, and test end-to-end data pipelines, including streaming, event, and batch ingestion, on cloud infrastructure using Azure and Cosmos DB.
Develop and implement ETL processes for data ingestion, transformation, and loading into data lakes.
Work extensively with Databricks and data warehousing concepts.
Design and develop custom, high-throughput, configurable frameworks and libraries.
Drive change through collaboration, influence, and demonstrations of proofs of concept.
Perform all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation.
Collaborate within an agile project team and follow best practices and coding standards.
Monitor and troubleshoot data pipelines to identify and resolve issues.
Implement data quality checks and validations to ensure data accuracy.
Perform data analysis and data exploration.
Continue to grow personal skillsets.

Benefits

Retirement savings plan (401(k)) with company match.
Insurance coverage, including basic life, medical, dental, vision, long-term disability, and optional additional coverage.
Paid time off, including vacation, sick leave, short-term disability, and leave for family care responsibilities.
Access to an Employee Assistance Program.
Incentive compensation, including eligibility for annual performance-based awards, subject to stated exclusions.
Eligibility for certain tax-advantaged savings plans.
Inclusive development opportunities.
Flexible work-life support.
Paid volunteer days.
Employee networks.