Data Engineer

Illumination Works LLC• US,

5d•Remote

About The Position

We are seeking an experienced Data Engineer to support challenging data engineering, data management, and analytics initiatives. In this role, you will develop and maintain data pipelines and data structures that enable advanced analytics and data science capabilities. You will work closely with data scientists and other technical stakeholders to understand evolving data needs, incorporate new data elements, and ensure reliable, high-quality data is available for analysis.

Requirements

4+ years of experience as a Data Engineer or in a similar data-focused role
Possess advanced SQL skills (i.e., complex joins, aggregations, window functions, transformations)
Proficiency with Python for data engineering and transformation
Hands-on experience with Databricks, including development and maintenance of notebooks and data pipelines
Experience developing and maintaining ETL/ELT pipelines and data integration workflows
Experience with Delta Lake
Experience with database design, data modeling, schema development, and data-centric documentation
Experience integrating and transforming data from multiple data source types
Experience troubleshooting data issues and optimizing queries and pipelines
Strong analytical and problem-solving skills
Excellent verbal and written communication skills and ability to collaborate effectively with technical and business stakeholders
Bachelor's degree in Computer Science, Mathematics, Data Science, Engineering, or a comparable academic discipline
Must hold an active Secret clearance with eligibility for Top Secret (U.S. Citizenship required)
Must successfully pass required background screening and drug testing

Nice To Haves

Experience supporting data scientists and advanced analytics or machine learning use cases
Functional knowledge of Department of Defense War or U.S. Air Force data sources and systems
Experience working with logistics, supply chain, maintenance, or other operational data
CompTIA Security+ certification
Holds Top Secret clearance

Responsibilities

Develop, maintain, and enhance data pipelines using Python, SQL, and Databricks
Develop and maintain Databricks notebooks supporting data ingestion, transformation, and delivery
Design and maintain data tables, schemas, and Delta Lake structures to support analytics and data science requirements
Work closely with data scientists to understand data requirements and modify pipelines, tables, and data structures as needs evolve
Develop complex SQL queries using joins, aggregations, window functions, and other advanced techniques to transform and prepare data
Integrate and transform data from multiple structured and semi-structured data sources
Support data quality through validation, troubleshooting, and resolution of data issues
Optimize data pipelines, SQL queries, and data structures for performance
Document data sources, transformations, data flows, schemas, and pipeline configurations
Collaborate with data scientists, engineers, analysts, and business stakeholders to deliver data solutions that support mission requirements