Data Engineer – Databricks, Python, and SQL

Codetech•Scottsdale, AZ

11h

About The Position

Develop data pipelines to ingest, load, and transform data from multiple sources. Use the Google Cloud-based Data Platform to design, optimize, deploy, and deliver data solutions supporting scientific discovery. Build cloud-based solutions in collaboration with digital product managers and deliver technology continuously in line with product roadmaps, using Agile and DevOps principles. Design, develop, test, and deploy ETL pipelines that extract data from various sources and transform it for target models. Support real-time and batch processing, data governance, and policies; maintain business contracts for data representation and storage; and ensure compliance with security, quality, regulatory, and data-governance requirements. Keep technical documentation current for lifecycle plans, audits, and reviews. Implement data movement and processing across systems and services such as S3, Snowflake, Kafka, and Spark. Work with Data Scientists, SREs, and Product Managers to deliver high-quality software that meets user requirements. Proactively experiment with new data-engineering tools and frameworks to drive improvement.

Requirements

Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
At least 5 years of experience as a data engineer building ETL/ELT data pipelines.
Experience with data-engineering best practices across the full software development lifecycle, including coding standards, code reviews, source control management, continuous integration, testing, and operations.
Experience with Python and SQL.
Experience with Agile, DevOps, automation of testing, builds and deployments, CI/CD, and Airflow.
Experience with Docker, Kubernetes, and shell scripting.
At least 2 years of experience with a public cloud such as AWS, Microsoft Azure, or Google Cloud.
At least 3 years of experience with distributed data or computing tools such as MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL.
At least 2 years of experience working on real-time data and streaming applications.
At least 2 years of experience with NoSQL implementations such as DynamoDB, MongoDB, Redis, or Elasticache.
At least 2 years of data-warehousing experience with tools such as Redshift, Snowflake, or Databricks.
At least 2 years of experience with UNIX/Linux, including basic commands and shell scripting.
Experience with visualization tools such as SSRS, Excel, PowerBI, Tableau, Google Looker, or Azure Synapse.

Nice To Haves

Experience with Java, C#, C++, Go, Ruby, or Rust.

Responsibilities

Develop data pipelines to ingest, load, and transform data from multiple sources.
Design, optimize, deploy, and deliver data solutions on the Google Cloud-based Data Platform to support scientific discovery.
Design and develop ETL programs to extract data from various sources and transform it to fit target models.
Test and deploy data pipelines to ensure compliance with data-governance and security policies.
Own real-time and batch processing implementations and support data governance and policies.
Maintain and enforce business contracts for how data should be represented and stored.
Ensure technical delivery complies with security, quality, and regulatory standards.
Keep relevant technical documentation up to date for lifecycle plans, audits, and reviews.
Implement ETL processes and move data between systems, including S3, Snowflake, Kafka, and Spark.
Collaborate with digital product managers, Data Scientists, SREs, and Product Managers to deliver robust, high-quality cloud-based solutions that meet user requirements.
Use Agile and DevOps principles to deliver technology solutions aligned with product roadmaps.
Experiment with data-engineering tools and frameworks to drive continuous improvement.