Data Engineer-(Microsoft Fabric, Spark/PySpark)

Computer Task Group, Inc•UNAVAILABLE, UNAVAILABLE

2d•Remote

About The Position

Join a dynamic data engineering team responsible for designing and delivering modern lakehouse solutions that support enterprise analytics and data-driven decision-making. This role focuses on building scalable data pipelines, implementing Microsoft Fabric lakehouse architectures, and developing Spark/PySpark solutions that integrate data from multiple enterprise systems. Candidates with Databricks or comparable lakehouse platform experience are also encouraged to apply.

Requirements

Strong experience with Microsoft Fabric, including: Lakehouse architecture and implementation, Spark Notebook development, Schema design and management, File and folder structure management, Security, permissions, and access management
Hands-on Spark or PySpark development experience.
Experience building and orchestrating enterprise data pipelines.
Experience integrating data from multiple source systems (10+ preferred).
Strong understanding of lakehouse data structures and data organization.
Excellent analytical, troubleshooting, and problem-solving skills.
5+ years of experience in Data Engineering, Data Warehousing, or related disciplines.
Proven experience designing and implementing scalable enterprise data platforms.
Strong hands-on experience with Spark/PySpark for large-scale data processing.
Experience developing modern lakehouse solutions using Microsoft Fabric, Databricks, or similar technologies.
Experience integrating and managing large, complex datasets from multiple enterprise systems.
Strong understanding of data modeling, ETL/ELT processes, and enterprise data architecture.
Experience working within Agile software development environments.
Excellent verbal and written English communication skills and the ability to interact professionally with a diverse group are required.

Nice To Haves

Databricks lakehouse engineering experience or experience with other modern lakehouse platforms.
Experience with cloud-based data engineering environments.
Jira, Kanban, and Agile methodologies.
Jenkins and CI/CD pipeline implementation.
Experience with enterprise data warehouse platforms such as Snowflake or Teradata.

Responsibilities

Design, develop, and support enterprise data engineering solutions using Microsoft Fabric.
Build and maintain scalable lakehouse architectures, including schema design and data organization.
Develop Spark Notebooks and Spark/PySpark applications for data ingestion, transformation, and processing.
Create and orchestrate reliable data pipelines integrating data from numerous enterprise source systems.
Design and manage file structures, metadata, and storage organization within lakehouse environments.
Implement and maintain security, permissions, and governance across Microsoft Fabric workspaces.
Optimize data processing performance and troubleshoot production issues.
Collaborate with architects, analysts, and business stakeholders to deliver high-quality data solutions.
Participate in Agile development processes, code reviews, testing, and CI/CD deployment activities.
Contribute to continuous improvement of data engineering standards and best practices.