Data Engineer

Sparibis•Washington, DC

13d•Remote

About The Position

Sparibis LLC is seeking a Data Engineer to provide expertise in the development, implementation, integration, and sustainment of data architectures, data hubs, data lakes, and data warehouse solutions. The role involves supporting the design and development of data models, data structures, and data acquisition processes to enable data-driven decision making across the HC/HR Data Domain. The Data Engineer will develop engineering and implementation plans for data hubs, data acquisition, data modeling, data integration, and related data engineering activities. This includes researching and evaluating existing data sources to identify authoritative sources for data hub development, planning and maintaining data architectures, and designing, developing, maintaining, and optimizing ETL/ELT data pipelines and data transformation processes. The position requires developing and maintaining batch and streaming data pipelines using technologies such as Spark, Python, Databricks, Palantir Foundry, Kafka, and related data engineering tools. Additionally, the role involves developing and maintaining data acquisition processes, implementing incremental data loading strategies, defining and implementing approaches for handling late-arriving data, and identifying opportunities to automate manual data processes. The Data Engineer will develop, maintain, and optimize data engineering solutions within Databricks and enterprise data lake environments, including configuring, monitoring, and managing Databricks clusters. Developing and maintaining Spark-based data processing solutions using PySpark, Spark SQL, Spark Data Frame, Data Sets, and related technologies, as well as implementing and maintaining Delta Lake solutions, is also a key responsibility. The role includes supporting data engineering and application development within Palantir Foundry, developing and maintaining streaming data solutions using Kafka, Kafka Streams, ksqlDB, and related technologies, and developing and maintaining Python-based data processing applications and AWS Lambda functions. Support for data integration and processing using AWS services such as S3, Kinesis, Lambda, and DynamoDB, along with monitoring, troubleshooting, and optimizing cloud-based data processing solutions, is required. The position also involves developing and maintaining data quality controls, validation processes, and data quality gates, developing and executing data-driven testing and unit testing, and establishing and maintaining data lifecycle policies and processes. Monitoring and troubleshooting data pipelines and processing jobs, and identifying opportunities to improve data processing performance, reliability, scalability, and maintainability are also key aspects of the role.

Requirements

5+ years of IT experience focusing on enterprise data engineering to include data modeling, data quality, data mapping tools, and other technical documentation.
Must be eligible to obtain and maintain a Secret security clearance.
United States Citizenship is required as part of the eligibility criteria to be able to obtain this type of security clearance.
Bachelor’s degree in computer science, Mathematics, Statistics or IT related field.
Professional experience in data architecture, data engineering, data hub, data lake, and/or data warehouse development.
Experience designing, developing, implementing, and supporting enterprise-scale data engineering solutions.
Experience with data modeling, data mapping, data quality, data integration, and data management.
Experience supporting large-scale, high-performance enterprise data applications.
Experience developing and supporting ETL/ELT data pipelines and data transformation processes.
Hands-on experience with Databricks required.
Experience with Python and SQL for data engineering and data processing.
Experience with Palantir Foundry, including building and maintaining ontologies, ETL/ELT pipelines, applications, and data-driven user interfaces preferred but not required.
Debug, troubleshooting, design and implement solutions to complex technical issues.
Ability to thrive in a team-based environment.
Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management.
Must have strong written and verbal communication.

Nice To Haves

Active CompTIA Security+ or CompTIA Network+ certification preferred.
Experience with Databricks and/or Palantir Foundry required.
If selected, the candidate must be able to obtain a CompTIA Security+ certification prior to being eligible to begin supporting the program.
This includes data integration tools such as SSIS, Pentaho, AWS Data Migration Service, etc.
Experience with Apache Spark and/or PySpark, include Spark SQL, Data Frames, and/or Data Sets preferred.
Experience supporting data integration, migration, transformation, data warehouse, data hub, and/or data lake/delta lake implementations.
Experience with batch and streaming data processing architectures.
Experience with Kafka and/or other distributed steaming technologies.
Experience developing and supporting Kafka Streams.
Experience working in AWS cloud environment with familiarity in AWS services such as S3, Lambda, Kinesis, and/or DynamoDB.
Experience with Git based version control, including branching, merging, pull requests, and repository management.
Experience supporting CI/CD pipelines and cloud monitoring with tools such as Jenkins, Docker, and/or CloudWatch preferred.
Experience with Palantir Foundry, including building and maintaining ontologies, ETL/ELT pipelines, applications, and data-driven user interfaces preferred.
Experience with the Jupiter data and analytics environment, particularly in support of the HC/HR Data Domain preferred but not required.

Responsibilities

Provide data engineering expertise in the development, implementation, integration, and sustainment of data architectures, data hubs, data lakes, and data warehouse solutions.
Support the design and development of data models, data structures, and data acquisition processes that enable data-driven decision making across the HC/HR Data Domain.
Develop engineering and implementation plans for data hubs, data acquisition, data modeling, data integration and related data engineering activities.
Research and evaluate existing data sources within the data lake and enterprise data environment to identify authoritative, reliable, and appropriate sources for data hub development.
Plan, create, and maintain data architectures, ensuring alignment with business requirements.
Design, develop, maintain, and optimize ETL/ELT data pipelines and data transformation processes supporting enterprise data integration and data hub activities.
Develop and maintain batch and streaming data pipelines using technologies such as Spark, Python, Databricks, Palantir Foundry, Kafka, and related data engineering tools.
Develop and maintain data acquisition processes to ingest, transform, validate, and integrate data from diverse enterprise sources.
Implement incremental data loading strategies to optimize data processing and data freshness.
Define and implement approaches for handling late-arriving data, processing windows, data freshness, and other data lifecycle considerations.
Identify opportunities to automate manual data processes and improve the efficiency, reliability, and scalability of data engineering workflows.
Develop, maintain, and optimize data engineering solutions within Databricks and enterprise data lake environments.
Configure, monitor, and manage Databricks clusters in accordance with DON policies, standards, security requirements, and technical guidelines.
Document Databricks cluster configurations, parameters, and operational standards for internal and external stakeholders.
Develop and maintain Spark-based data processing solutions using PySpark, Spark SQL, Spark Data Frame, Data Sets, and related technologies.
Implement and maintain Delta Lake solutions, including Delta Live Tables where applicable, to support scalable and reliable data processing.
Optimize data processing, storage, and query performance within data lake environments.
Support data engineering and application development activities within Palantir Foundry, including development and maintenance of ontologies, ETL/ELT pipelines, applications, and data-driven user interfaces.
Support the integration of Palantir Foundry capabilities with enterprise data engineering and analytics environments.
Develop and maintain streaming data solutions using Kafka, Kafka Streams, ksqlDB, and related technologies.
Configure and manage Kafka topics and associated components, including Schema Registry, to support reliable and scalable data processing.
Develop and maintain Python-based data processing applications and AWS Lambda functions.
Support data integration and processing using AWS services such as S3, Kinesis, Lambda, and DynamoDB.
Monitor, troubleshoot, and optimize cloud-based data processing solutions for performance and scalability.
Develop and maintain data quality controls, validation processes, and data quality gates to ensure data accuracy, completeness, consistency, and reliability.
Develop and execute data-driven testing and unit testing for Spark, Python, and other data processing solutions.
Establish and maintain data lifecycle policies and processes, including retention, backup, recovery, and data management requirements.
Monitor and troubleshoot data pipelines and processing jobs to identify and resolve data quality, performance, and integration issues.
Identify opportunities to improve data processing performance, reliability, scalability, and maintainability.