Senior Data Engineer

Waverly Advisors, LLC•Birmingham, AL

9h•Onsite

About The Position

Waverly Advisors is seeking a Senior Data Engineer to join their team in Birmingham, AL. This role is essential to the company's mission of serving clients and driving the firm forward. The Senior Data Engineer will play a key part in designing how data is ingested, validated, modeled, and consumed by the rest of the firm. The company is building a modern data platform on Databricks, integrating various data sources into a governed source of truth for BI, internal applications, and AI workloads. All deployments are managed as code and adhere to the standards of a regulated financial services organization. The ideal candidate will extend the existing ingestion framework, making data onboarding more efficient and configuration-driven.

Requirements

Production-depth experience with Spark and Delta Lake, including Structured Streaming, Auto Loader, schema evolution, and idempotent pipeline design.
Hands-on Databricks experience, including jobs, serverless compute, and working within Unity Catalog as a data producer: schema and table design, tagging, row filters and column masks, and grants managed as code.
Databricks Asset Bundles or equivalent deployment tooling. Our jobs and pipelines are deployed as code through CI, not built in the workspace UI.
Experience building or substantially owning a framework, not just a collection of pipelines.
Real API integration experience, including rate limits, inconsistent pagination, expiring credentials, and vendor behavior that changes without notice.
Testable Python, with transform logic structured as functions and pytest as part of your normal workflow.
Strong SQL skills, including window functions and modeling patterns for slowly changing data.
Comfort inheriting an existing codebase, learning its conventions, and improving it.
Sound judgment about which technical decisions are easy to reverse and which ones are long-term commitments.
Domain knowledge is a real advantage. If you have worked with custodial data, portfolio accounting, or trust and entity structures, tell us. If you came to engineering through the industry rather than the other way around, we are interested, provided the engineering depth above is genuinely there.

Nice To Haves

Lakeflow or Delta Live Tables for declarative pipelines.
Azure services as a consumer: ADLS storage paths, Key Vault-backed secret scopes, and Event Hubs for streaming sources.
Data modeling for financial or accounting systems, including entity, ownership, and hierarchy structures.
Reverse ETL or writing data back into operational business systems.
Experience in financial services or another regulated industry.

Responsibilities

Extend our ingestion framework across data contracts, schema validation, quarantine handling, freshness monitoring, and provenance.
Onboard data sources end to end, from contract authoring through production go-live and monitoring.
Design the master account, entity, and household model that conforms data from every source system.
Build the curated datasets the firm runs on, from conformed core tables through the reporting and analytics layers that BI, internal applications, and AI consume.
Set testing standards for data code, including unit tests, contract fixtures, and integration tests on masked data.
Build production data operations, including quarantine triage, reprocessing workflows, and data quality expectations.
Write the runbooks and decision records that let the next engineer pick up your work and let an examiner trace a number back to its source file.