Data Engineer

Persistent Systems•New York, NY

1d•$133,000 - $166,000•Onsite

About The Position

Persistent Systems is seeking a Data Engineer to join its team at its New York City headquarters. The role designs, builds, and operates pipelines, lakehouses, and data models that provide trusted, well-governed data for analytics, reporting, and AI solutions, with a primary focus on Microsoft Fabric and the Azure data platform. The engineer connects operational, file, API, and streaming sources to downstream users and applications, transforming data through medallion layers in OneLake and publishing curated, secure datasets for Power BI, data science, and retrieval-augmented generation (RAG) workloads. The role may also integrate tools from other providers where required. The position owns data engineering work from source-system discovery and requirements through design, development, testing, deployment, monitoring, and ongoing support. It collaborates with analysts, BI developers, AI engineers, architects, security teams, and business stakeholders to establish data requirements, freshness targets, and acceptance criteria. The engineer develops reusable ingestion and transformation patterns, data contracts, and consistent lineage, while documenting designs and operating procedures and contributing to peer reviews and engineering discipline. The work includes implementing Microsoft Fabric Lakehouse and Warehouse architectures on OneLake; building batch, incremental, and streaming ingestion; developing transformations in PySpark, Spark SQL, T-SQL, and KQL; designing dimensional models and supporting Power BI semantic models; and implementing data quality, security, governance, and observability. The role also manages version control, CI/CD, infrastructure provisioning, AI-oriented data preparation, and performance and cost optimization. The total compensation range is $133,000 to $166,000 per year based on individual qualifications, with eligibility for a yearly discretionary bonus.

Requirements

3–5 years of professional experience in data engineering, ETL/ELT development, or data warehousing, including building and supporting production data pipelines.
Bachelor’s degree in computer science, information systems, engineering, mathematics, or a related field, or an equivalent combination of education and relevant professional experience.
Advanced SQL proficiency, including complex joins, common table expressions, window functions, MERGE and upsert logic, query tuning, and validation against source systems.
Strong Python skills for data engineering, including PySpark or Spark SQL for distributed data processing.
Hands-on experience building pipelines with Fabric Data Factory, Azure Data Factory, or Azure Databricks, including parameterization, scheduling, error handling, and incremental loads.
Experience designing lakehouse or data warehouse solutions using Delta Lake or comparable table formats, medallion architecture, and dimensional modeling.
Working knowledge of Azure storage, identity, networking, and RBAC, including secure data access with Microsoft Entra ID.
Experience with Git-based version control and CI/CD practices for data workloads, including branching, code review, and environment promotion.
Demonstrated ability to troubleshoot pipeline failures and data quality issues, communicate clearly, and document data flows and lineage.

Nice To Haves

Production experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, mirroring, shortcuts, and Fabric deployment pipelines.
Experience with real-time data using Fabric Real-Time Intelligence, Eventstreams, Eventhouse and KQL, Azure Event Hubs, or Spark Structured Streaming.
Experience with Azure Databricks, Unity Catalog, and Lakeflow, including integration with Microsoft Fabric.
Experience building release pipelines in Azure DevOps (Azure Repos and Azure Pipelines) and provisioning data infrastructure with Bicep or Terraform.
Experience with Microsoft Purview data governance, including data cataloging, lineage, sensitivity labels, and data quality.
Experience preparing data for AI workloads, including Azure AI Search indexing, Microsoft Foundry solutions, or RAG pipelines.
Experience with complementary non-Microsoft tools such as dbt, Apache Airflow, Apache Kafka, or Snowflake.
Experience with Microsoft Power Platform, including Power Apps, Power Automate, and Dataverse, and integrating low-code solutions with governed enterprise data.
Experience using AI coding agents such as Claude Code or Codex to accelerate pipeline development safely.
Relevant certifications or credentials, such as Microsoft Certified: Fabric Data Engineer Associate (DP-700), Microsoft Certified: Azure Databricks Data Engineer Associate (DP-750), Microsoft Certified: Fabric Analytics Engineer Associate (DP-600), Microsoft Certified: Power Platform Fundamentals (PL-900), or Databricks Certified Data Engineer Associate or Professional.
Experience operating in a regulated, security-sensitive, government, or defense-adjacent environment.

Responsibilities

Own data engineering work from source-system discovery and requirements through design, build, testing, deployment, monitoring, and ongoing support.
Partner with analysts, BI developers, AI engineers, architects, security, and business stakeholders to define data requirements, freshness targets, and acceptance criteria.
Build reusable ingestion patterns, transformation frameworks, and data contracts to make pipelines, definitions, and lineage consistent and maintainable.
Evaluate Microsoft Fabric, Azure, and complementary tools against requirements for scale, performance, security, and total cost of ownership, and track platform changes and new features.
Document designs, data flows, and operating procedures; review peers’ work; and contribute to engineering discipline.
Design and implement bronze, silver, and gold architectures in Microsoft Fabric Lakehouse and Warehouse on OneLake using Delta Lake tables, OneLake shortcuts, and appropriate partitioning and table optimization.
Build batch and incremental ingestion with Fabric Data Factory pipelines, Copy job, Dataflow Gen2, and Azure Data Factory, and replicate operational databases into OneLake using mirroring and change data capture.
Develop transformations in Spark notebooks using PySpark and Spark SQL, T-SQL, and KQL; handle duplicate, missing, and late-arriving data; and implement slowly changing dimensions and incremental merge patterns.
Build and maintain Azure Databricks pipelines using Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Auto Loader, and Unity Catalog, integrating them with OneLake and Azure Data Lake Storage Gen2.
Ingest and process streaming data with Fabric Real-Time Intelligence, Azure Event Hubs, and Spark Structured Streaming.
Design star-schema fact and dimension models and collaborate with BI developers on Power BI semantic models, including Direct Lake.
Implement validation rules, schema enforcement, reconciliations, and automated data quality tests; monitor pipelines and resolve failures through root-cause analysis.
Apply OneLake security, row- and column-level security, dynamic data masking, sensitivity labels, and Microsoft Purview cataloging and lineage; secure access using Microsoft Entra ID, managed identities, and Azure Key Vault.
Manage notebooks, pipelines, and SQL database projects in Git-based version control; promote changes through Fabric deployment pipelines and Azure Pipelines with automated tests and approvals; and provision Azure resources with Bicep or Terraform.
Prepare curated datasets and document-processing pipelines for Azure AI Search, Microsoft Foundry, and Copilot solutions, including chunking, metadata enrichment, and freshness monitoring for RAG.
Tune Spark jobs, Delta tables, warehouse queries, and pipeline schedules, and monitor Fabric capacity consumption and Azure spend.