Staff Data Engineer

Unite Us•New York, NY

7d•$175,000 - $200,000•Remote

About The Position

The Staff Data Engineer will lead enterprise-wide data architecture strategy and long-term data platform design. Evaluate and approve major technical designs, including establishing engineering best practices. Serve as technical advisor to leadership and influence product/data roadmap. Champion data governance, democratization, and scalable data standards. Work with cross functional teams including the Solutions Delivery, Business Intelligence; Predictive Analytics, and Enterprise Services teams to assist with and support any data-related technical issues and/or needs. Analyze, manipulate, and process large sets of data using statistical software. Develop ETL (Extract, Transform, Load) and data pipeline solutions using Airflow, DBT, Python, and Snowflake to load a data warehouse. Design and manage ETL workflows to support large-scale ingestion, transformation, and loading of data. Build and scale CI/CD pipelines and enterprise-wide data quality frameworks. Test internal data pipelines for reliability and performance. Test, validate, and reformulate models to ensure accuracy. Complete and improve monitoring and validation processes to maintain pipeline reliability and data accuracy. Write new functions or applications in programming languages to conduct analysis. Build streaming pipelines with Confluent Kafka for event-driven processing. Build dashboards and monitors for pipeline health, freshness, uptime, and SLA compliance. Apply data modeling to extract and analyze information from large structured and unstructured datasets.

Requirements

Developing and optimizing scalable pipelines using Apache Airflow and Python
Building streaming pipelines using Kafka
Tuning big data systems using Kafka and Hive
Using Spark to create jobs for batch and streaming analytics
Writing and using complex SQL queries to retrieve data from data warehouses for analysis and reporting
Using AWS for data processing and large-volume workloads
Automating deployment of code and processes using Python
Working with cloud-based infrastructure
Building Datadog dashboards to monitor pipelines and system performance
Using Docker and Kubernetes to improve and validate containerization and automation reliability
Implementing Continuous Integration/Continuous Deployment (CI/CD) using Jenkins
Utilizing GitHub as a code repository
Utilizing Windows and Linux operating systems

Responsibilities

Lead enterprise-wide data architecture strategy and long-term data platform design.
Evaluate and approve major technical designs, including establishing engineering best practices.
Serve as technical advisor to leadership and influence product/data roadmap.
Champion data governance, democratization, and scalable data standards.
Work with cross functional teams including the Solutions Delivery, Business Intelligence; Predictive Analytics, and Enterprise Services teams to assist with and support any data-related technical issues and/or needs.
Analyze, manipulate, and process large sets of data using statistical software.
Develop ETL (Extract, Transform, Load) and data pipeline solutions using Airflow, DBT, Python, and Snowflake to load a data warehouse.
Design and manage ETL workflows to support large-scale ingestion, transformation, and loading of data.
Build and scale CI/CD pipelines and enterprise-wide data quality frameworks.
Test internal data pipelines for reliability and performance.
Test, validate, and reformulate models to ensure accuracy.
Complete and improve monitoring and validation processes to maintain pipeline reliability and data accuracy.
Write new functions or applications in programming languages to conduct analysis.
Build streaming pipelines with Confluent Kafka for event-driven processing.
Build dashboards and monitors for pipeline health, freshness, uptime, and SLA compliance.
Apply data modeling to extract and analyze information from large structured and unstructured datasets.