[C3A] Senior Data Platform Engineer (Microsoft Fabric)

Software Mind

3d

About The Position

Software Mind is seeking a Senior Data Platform Engineer for a client building a next-generation risk analysis and scoring platform on Microsoft Fabric. The role owns the medallion data architecture end to end and contributes to both the platform foundation and risk product delivery, working with a Senior Technical Lead, a Senior DevOps Engineer, and a mid-level developer.

The platform ingests transaction and merchant data through SFTP and APIs, tokenizes sensitive PAN data at ingestion, and lands canonical Silver data in Microsoft Fabric. The engineer will build pipelines that populate canonical Silver, consume it through OneLake shortcuts, and materialize risk-specific Gold tables for custom DMN rules and pre-scored Mastercard Brighterion feeds.

The work includes designing Bronze, Silver, and Gold Delta tables and their physical layouts; implementing Fabric Spark feature pipelines and Direct Lake semantic models; building adapters and idempotent MERGE pipelines; configuring OneLake and workspace layouts; and optimizing Fabric capacity. The role also partners with DevOps on infrastructure as code and may consult on a ClickHouse contingency if Direct Lake latency is insufficient for client-tier UIs.

Requirements

• At least 5 years of data engineering experience.
• At least 2 years of hands-on Spark development; PySpark preferred.
• Deep knowledge of Delta Lake internals, including transaction logs, checkpoints, retention settings, VACUUM, OPTIMIZE, deletion vectors, and statistics collection.
• Expertise in Parquet, including footers, statistics allowlists, row-group sizing, dictionary encoding, and column indexes.
• Hands-on Microsoft Fabric experience with workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, and the SQL analytics endpoint.
• SQL performance tuning skills for OLAP workloads in Fabric and OLTP workloads in PostgreSQL.
• Strong Python data engineering skills, including pandas, PyArrow, and delta-rs.
• Ability to communicate architecture clearly in writing and work autonomously on well-scoped features.

Nice To Haves

• PCI-scoped data platform experience.
• Financial services or payments domain background.
• Power BI semantic model authoring with DAX.
• Familiarity with published data-standard governance patterns, including logical model versus physical profile, SCD2 dimensions, natural-key idempotency, and extension attribute stores.
• Experience with OneLake shortcuts and cross-workspace lakehouse patterns.
• ClickHouse or similar OLAP engine familiarity.
• Terraform experience for Azure data resources.
• Prior experience with fluid or schema-flexible ingestion patterns.

Responsibilities

• Design and implement Bronze, canonical Silver, and Gold Delta table schemas with PAN-safe physical layouts, including statistics allowlists, liquid clustering, retention settings, and V-Order optimization.
• Build Fabric Spark notebooks for rolling transaction and merchant features, hierarchical rollups, and cross-merchant features using pan_token joins.
• Build per-flavor-version Silver adapters, unioned by name and tested against golden fixtures.
• Build the idempotent Bronze-to-canonical-Silver MERGE DAG using _natural_key and Aperia's canonical Fabric internal specification v1.4.
• Build Gold materialization pipelines for transaction-risk and merchant-risk grains, with correct score-source stamping and idempotent merge semantics.
• Design and implement Direct Lake semantic models for reporting surfaces and performance-test them against representative data volumes.
• Configure OneLake namespace layouts, workspace catalogs, and cross-workspace shortcut relationships with the residuals programme.
• Establish idempotent Delta write patterns using delta-rs from Python containers, with segmented parallel writes and a single atomic commit.
• Size Fabric capacity, select F-SKUs, conduct load testing, and make Reservation-versus-PAYG decisions.
• Partner with the DevOps engineer on Fabric infrastructure as code, including Terraform capacity provisioning and Fabric REST workspace bootstrap.
• Consult on a ClickHouse contingency if Direct Lake latency is insufficient for client-tier UIs.

Benefits

• Flexible and remote cooperation options.
• International projects with leading global clients.
• Travel related to international projects.
• Non-corporate atmosphere.
• Access to language classes.
• Access to knowledge-sharing initiatives.
• Access to private healthcare and life insurance.
• Access to a multisport card.
• Availability and terms of individual benefits may vary depending on the chosen form of cooperation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed