Sr. Data Engineer

ARC-One Solutions•,

2d•$99,937 - $157,044•Remote

About The Position

Manages and evolves the enterprise data lake and data warehouse while ensuring the reliable, secure, and efficient flow of high-quality data. Implements data processes, managing data architecture, designing ETL processes, and analyzing data for business insights.

Requirements

Bachelor's degree in a computer-related field from an accredited college or university and five (5) or more years of experience in data engineering, building scalable and distributed ETL data pipelines in enterprise environments.
Experience building and operating scalable AWS-based data platforms and pipelines using services including Lambda, Glue, Athena, S3, Redshift, DMS, MWAA (Airflow), and Step Functions, supporting batch, CDC, and near real-time data processing.
Advanced proficiency in Python, SQL, and PySpark with hands-on experience developing reusable ETL/ELT frameworks, data warehouses, data marts, and integrations across databases, APIs, event streams, and analytics environments.
Experience implementing data quality, governance, and optimization best practices, including automated validation frameworks, Lake Formation and Glue Data Catalog, performance tuning, and cost optimization across AWS data services.
Strong communication skills with the ability to translate complex data concepts for business stakeholders; experience in healthcare, life sciences, and other highly regulated environments with HIPAA, GDPR, FDA, or similar compliance requirements preferred.
Experience with metadata management, data lineage, data observability, master data management, or enterprise data catalog solutions.
Knowledge with data modeling & analytical data model, schema design, schema evolution, and data structure optimized for reporting and analytics.
Knowledge of data lake and data warehouse architecture include data partitioning and columnar storage format such as Parquet.
Relevant AWS certification, such as AWS Certified Data Engineer – Associate, or an equivalent cloud or data engineering certification.

Nice To Haves

experience in healthcare, life sciences, and other highly regulated environments with HIPAA, GDPR, FDA, or similar compliance requirements

Responsibilities

Design, implement and maintain scalable data pipnes on WS using S3, DMS, Glue, lambda, step function/MWAA & Redshift.
Develop robust batch and near-real-time ETL/ELT workflow to ingest, cleanse, transform and load data from databases, legacy applications and event streams using Python & Pyspark.
Design incremental/CDC mechanism, including restart ability, idempotency, duplicate handling and recovery.
Implement automated controls for completeness, accuracy, reconciliation, schema changes & lineage.
Optimize Glue/Spark, Athena, Redshift & S3 workload through partitioning, columnar formats, query tuning and appropriate storage/compute design.
Design near real time/event-driven pipelines using Kinesis/Kafka where required, covering ordering, retry, idempotency and failure recovery.
Implement AWS data security, least privilege access, data classification and governance controls.
Monitor pipelines such as CloudWatch, troubleshoot failure and resolving production data incidents.
Enforce Git/version control, code review, automated testing and CI/CD practices.
Work with product owners, architect, reporting and business stakeholders to translate requirements into scalable data solutions.
Document pipelines and operational procedures.