Data Engineer

Peter Millar • Durham, NC

1d

About The Position

We are seeking a Data Engineer to design, build, and support modern data pipelines and data platform solutions using Microsoft and cloud-based technologies. This role will be responsible for developing and optimizing scalable data pipelines, supporting the Microsoft Fabric platform, and ensuring high-quality, reliable data is available for reporting, analytics, and business operations. The ideal candidate has strong experience with SQL, Azure data services, and ETL/ELT development, and is comfortable working across both pipeline development and data platform responsibilities.

Requirements

• Bachelor’s degree in Computer Science, Information Systems, Data Engineering, or related field (or equivalent experience).
• 3–5 years of hands-on experience in data engineering or a related technical role.
• Strong SQL skills, including query optimization, indexing, and performance tuning.
• Proficiency in Python and/or PowerShell for data transformation and automation.
• Hands-on experience with Azure data services, including Data Factory, Synapse Analytics, Azure SQL, and Data Lake Storage.
• Working experience with Microsoft Fabric, including Lakehouse, Dataflows, and pipelines.
• Solid understanding of ETL/ELT design patterns and data pipeline best practices.
• Experience working with structured and semi-structured data formats (JSON, XML, etc.).
• Familiarity with source control tools such as Git/GitHub and CI/CD practices.
• Strong analytical, problem-solving, and communication skills.

Nice To Haves

• Experience with Spark (PySpark) for large-scale data processing.
• Familiarity with Delta Lake and Parquet data formats.
• Exposure to Power BI or other data visualization tools.
• Experience with APIs, including development and integration.
• Microsoft Fabric certification (DP-600) or interest in pursuing certification.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Azure Data Factory, Microsoft Fabric Data Factory, and Dataflows.
• Develop and optimize data ingestion workflows across structured and semi-structured data sources.
• Troubleshoot pipeline failures, data issues, and performance bottlenecks to ensure reliable data delivery.
• Support and maintain data platform components within Microsoft Fabric, including Lakehouse, Warehouse, and OneLake.
• Work with medallion architecture (bronze, silver, gold) to structure and manage data transformations.
• Contribute to Spark and notebook-based data processing for large-scale data transformation.
• Monitor data pipelines and platform health, and support alerting and incident response processes.
• Perform data validation, profiling, and cleansing to ensure high-quality and accurate data.
• Work with business stakeholders to gather data requirements and translate them into technical solutions.
• Collaborate with analysts, data scientists, and application teams to support reporting and downstream use cases.
• Write and maintain documentation for data pipelines, transformations, and workflows.
• Participate in code reviews and contribute to CI/CD and DevOps practices using Git/GitHub.
• Follow and help enforce data governance, security, and best practices across the data platform.

Benefits

• equal employment opportunities

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level