Senior Data Engineer

Commercial Bank of California•United State, US,

15h•$135,000 - $170,000

About The Position

The Senior Data Engineer designs, builds, and operates scalable, secure data platforms that support enterprise analytics, reporting, and advanced data use cases, including financial services and payments data. The role calls for 7–10+ years of hands-on data engineering experience and combines data engineering with architectural ownership of end-to-end pipelines, data models, and analytical data stores across the Azure ecosystem. The engineer leads implementation of Microsoft Fabric, Azure Databricks, and modern lakehouse architectures, including migrating Databricks workloads to Microsoft Fabric where appropriate. The role partners with business intelligence, analytics, AI, and business stakeholders, with emphasis on data reliability, governance, and performance at scale to support operational and strategic decision-making.

Requirements

7–10+ years of professional experience in data engineering, data platform development, or analytics engineering.
Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.
Deep hands-on experience with Microsoft Fabric, including Lakehouse, Data Pipelines, OneLake, and Semantic Models.
Deep hands-on experience with Azure Databricks, including Spark, Delta Lake, and performance tuning.
Experience with data modeling for analytics and BI.
Experience with Azure DevOps, CI/CD, Git, and automated deployments.
Experience with business intelligence platforms, especially Power BI.
Microsoft data and analytics stack expertise, including Fabric, Azure Data Lake, Synapse, Power BI, Microsoft Purview, and Entra ID.
Databricks expertise, including Unity Catalog, Delta Live Tables/Lakeflow, Databricks Workflows, and cluster and cost optimization.
Advanced SQL proficiency and strong experience with Python or Spark-based development.
Proven experience building and supporting production-grade data pipelines at scale.

Responsibilities

Design, build, and maintain enterprise-grade data platforms using Microsoft Fabric, Azure Data Lake, Azure Synapse, and Azure Databricks.
Architect and implement lakehouse and medallion architectures (Bronze/Silver/Gold) for analytics and BI workloads.
Define standards for data ingestion, transformation, storage, and access across structured and semi-structured data sources.
Plan and execute Databricks-to-Fabric migrations, including workload assessment, Delta/OneLake data movement, notebook and pipeline conversion, semantic model re-pointing, validation, and cutover.
Develop and optimize batch and streaming data pipelines using Databricks, Fabric Data Pipelines, and Azure-native services.
Implement ETL/ELT processes and ensure data quality, accuracy, and consistency across systems.
Monitor, tune, and optimize data pipelines for performance, scalability, and cost efficiency.
Integrate and model FIS core banking and H360 system data in the lakehouse for analytics and reporting.
Build and support payments data pipelines for ACH, wire, card, and real-time payments, including ingestion, reconciliation, and reporting.
Design and maintain logical and physical data models optimized for analytics, reporting, and downstream consumption.
Build semantic models and curated datasets for Power BI and enterprise BI use cases.
Partner with BI developers, analysts, and business stakeholders to turn reporting requirements into scalable data solutions.
Implement and manage Azure DevOps CI/CD pipelines for data engineering workloads.
Enforce source control, automated testing, deployment, and environment-management best practices.
Ensure high availability, fault tolerance, and operational excellence for production data systems.
Implement data governance, security, and access controls, including role-based access and data classification.
Ensure compliance with enterprise data policies and regulatory requirements.
Collaborate with security and infrastructure teams to enforce secure data handling and platform standards.
Validate daily, weekly, and monthly data loads; investigate mismatches; reconcile source-to-target records; validate business-critical reporting datasets; and verify data completeness, accuracy, and timeliness.
Mentor data engineers and analytics teams, provide architectural guidance and code reviews, and lead best practices across data initiatives.
Partner with business, analytics, and AI teams to enable data-driven decision-making.