Databricks Engineer

Treliant LLC•New York, NY

2d•$160,000 - $200,000•Hybrid

About The Position

Treliant is seeking two Databricks Engineers to join an existing team at a large, global bank. This is a hybrid role located in the Greater New York City metro area. The successful candidates will work with a mix of bank and Treliant staff. The client bank is centralizing its key reporting and aggregation processes on cloud platforms, with a focus in 2024 on centralizing cashflow aggregation onto a Databricks-based platform (Databricks and Deltalake). The project involves collecting actual and forecast cashflows from real-time and batch data sources, cleaning and enriching the data, and storing the aggregated data in Deltalake for reporting purposes. This role will also involve guiding the migration, cost management, use case prioritization, and communicating with data providers and consumers to understand requirements and map data fields.

Requirements

Played an important role in at least one Databricks platform implementation.
Familiarity with ETL/ELT in Databricks, Medallion architecture, and Deltalake.
Familiarity with Streaming technologies such as Kafka.
Familiarity with Delta Share and Deltalake Connector.
Proficiency in SQL, Python and PySpark.
Background in financial products and processes.
Domain knowledge in cashflow forecasting and liquidity management.
Domain knowledge in financial products including securities such as swaps, and banking products such as loans.
Domain knowledge in reference data including counterparty and business hierarchy data.
Domain knowledge in regulatory and risk management reporting needs.
Experience of the challenges of aggregating data in bulk across business units and asset classes.
Element of communication and leadership, rather than being purely a technical role.
Experience in data migration, data integration, and delivery of data warehouses is also advantageous.

Responsibilities

Centralizing cashflow aggregation onto a platform based on Databricks technologies (Databricks and Deltalake).
Collecting both actual and forecast cashflows from both real-time and batch data sources.
Cleaning and enriching the data.
Storing the resulting aggregated data in Deltalake for use in reporting.
Guiding and understanding aspects of the migration such as cost management and use case prioritization.
Communicating with data providers and data consumers to understand the requirements and to source and map the needed data fields.