Sr. Databricks Data Engineer, Vice President

State Street • Princeton, NJ

10d • $120,000 - $202,500 • Hybrid

About The Position

The Sr. Databricks Data Engineer will contribute hands-on to developing, enhancing, and maintaining platforms as a member of an agile scrum team. The role involves designing, building, and testing end-to-end data pipelines on cloud-based infrastructure, developing ETL processes, and working extensively with Databricks and data warehousing. The engineer will also develop custom high-throughput, configurable frameworks and libraries; collaborate to drive change through proofs of concept; and contribute across the software development lifecycle, from design and coding through integration testing, deployment, and documentation. The position is onsite four days per week and remote one day per week, with standard business hours aligned to project and stakeholder needs (8 a.m. to 5 p.m.).

Requirements

• Bachelor’s degree in a computer- or IT-related subject.
• 10+ years of overall Big Data data pipeline experience.
• 5+ years of hands-on Databricks experience.
• 5+ years of cloud-based development experience, including Azure services, Azure DevOps, Kubernetes, and Docker.
• Hands-on development experience with Databricks SQL and Scala.
• Proficient working knowledge of data warehousing platforms and technologies such as Databricks, Delta Lake, and Apache Spark.
• Strong understanding of the Databricks platform, including clusters, jobs, and other resources.
• Experience troubleshooting DevOps pipelines, including ADO and Harness.
• Experience in a multi-developer environment using version control such as Git.
• Experience handling semi-structured data such as Avro, JSON, and XML.
• Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer.
• Strong critical-thinking, communication, and problem-solving skills.
• Ability to communicate effectively and professionally, both orally and in writing.
• Team-oriented approach, positive attitude, enthusiasm, initiative, and self-motivation.
• Ability to multitask; energetic, fast learner, and problem solver.
• Experience working in the financial industry.
• Experience with agile development methodology.

Nice To Haves

• Experience in microservices architecture and understanding of cloud computing, preferably Azure.
• Experience with Snowflake.

Responsibilities

• Design, build, and test end-to-end data pipelines for streaming, events, and batch ingestion on cloud-based infrastructure using Azure and Cosmos DB.
• Develop and implement ETL processes to ingest, transform, and load data into data lakes.
• Work with Databricks and data warehousing concepts.
• Design and develop custom high-throughput, configurable frameworks and libraries.
• Drive change through collaboration, influence, and demonstration of proofs of concept.
• Contribute to the software development lifecycle, including design, coding, integration testing, deployment, and documentation.
• Work collaboratively within an agile project team and follow best practices and coding standards.
• Monitor and troubleshoot data pipelines to identify and resolve issues.
• Implement data quality checks and validations to ensure data accuracy.
• Perform data analysis and data exploration.
• Grow personal skillsets.

Benefits

• 401(k) retirement savings plan with company match.
• Insurance coverage including basic life, medical, dental, vision, and long-term disability, as well as optional additional coverages.
• Paid time off, including vacation, sick leave, short-term disability, and leave for family care responsibilities.
• Access to an Employee Assistance Program.
• Eligibility for annual performance-based incentive awards, with some sales-role exceptions.
• Eligibility for certain tax-advantaged savings plans.
• Inclusive development opportunities, flexible work-life support, and paid volunteer days.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior