Data Engineer

CosmoProf•Denton, TX

17d•Hybrid

About The Position

The Data Engineer is responsible for supporting the design, development, and maintenance of enterprise data solutions on the Data Bricks Lake House Platform. This role works closely with senior Data Engineers, Data Architects, Analytics Engineers, and business stakeholders to develop data pipelines, transform and validate data, and support reporting, analytics, artificial intelligence (AI), and machine learning (ML) initiatives. The Data Engineer I will participate in the implementation of modern cloud-based data solutions while developing technical expertise in data engineering best practices.

Requirements

Bachelor’s degree in computer science, Information Systems, Software Engineering, Data Engineering, or a related technical field.
3+ years of experience in Data Engineering or Software Engineering.
1+ years of hands-on experience with the Databricks Lakehouse Platform.
Experience with Azure Data Lake Storage (ADLS), Amazon S3, or similar cloud storage platforms.
Experience with Microsoft Azure, AWS, or Google Cloud Platform (Azure preferred).
Experience supporting enterprise analytics platforms such as Power BI.
Retail, merchandising, supply chain, or eCommerce experience preferred.

Nice To Haves

Databricks Lakehouse Platform
Delta Lake and Delta Live Tables (DLT)
Unity Catalog
Medallion Architecture (Bronze, Silver, Gold)
ETL/ELT Pipeline Development
Data Modeling and Data Warehousing
Lakehouse Architecture
SQL Development and Performance Optimization
Python Programming
REST API and Enterprise System Integration
Azure Data Lake Storage (ADLS) or Cloud Object Storage
Azure, AWS, or Google Cloud Platform
Business Intelligence and Analytics Support
Analytical and Problem-Solving Skills
Strong Written and Verbal Communication
Cross-Functional Collaboration
Planning, Organization, and Time Management
Results-Oriented with Strong Attention to Detail

Responsibilities

Design, develop, and maintain scalable data pipelines using the Databricks Lakehouse Platform.
Build ETL/ELT processes utilizing Apache Spark, PySpark, Spark SQL, and Delta Lake.
Design and implement Medallion Architecture (Bronze, Silver, Gold).
Develop batch and streaming data ingestion pipelines from enterprise systems, APIs, and cloud platforms.
Build reusable data transformation frameworks using PySpark and SQL.
Optimize Spark jobs and Delta Lake tables for performance, scalability, and cost efficiency.
Configure and support Unity Catalog for data governance, security, and lineage.
Develop workflows using Databricks Workflows and Jobs.
Support Change Data Capture (CDC) and incremental data processing.
Build semantic data models supporting reporting, analytics, AI, and machine learning.
Implement automated testing, monitoring, and alerting for data pipelines.
Create technical documentation, metadata, and operational standards.
Research and recommend new technologies and platform enhancements