Data Engineer - Databricks
About The Position
Steampunk is seeking a seasoned Data Engineer to work with its team and clients to develop enterprise-grade data platforms, services, and pipelines. The role calls for a technologist with strong communication and customer service skills and a passion for data and problem solving. The position focuses on architecting reliable, high-performance data pipelines and ingestion patterns, assessing existing ETL jobs and analytics workflows, answering technical questions about data products, and developing cloud database and data warehouse solutions. The preferred cloud environment is AWS, with Azure and GCP as alternatives. The engineer will support an Agile software development lifecycle and contribute to the growth of Steampunk’s Data Exploitation Practice.
Requirements
- Experience supporting ICE contract work.
- Experience working in Design Intelligence.
- Bachelor’s degree and at least 6 years of total experience, or equivalent experience and education.
- At least 6 years of direct Data Engineering experience.
- Experience with big data tools such as Hadoop, Spark, and Kafka.
- Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
- Experience with data pipeline and workflow management tools such as Azkaban, Luigi, and Airflow.
- Experience with AWS cloud services, including EC2, S3, RDS, Glue, Step Functions, Lambda, EMR, DynamoDB, DocumentDB, Redshift, Aurora, and Athena.
- Experience with data platforms such as Databricks and Snowflake.
- Experience with data streaming systems such as Kafka, Storm, and Spark Streaming.
- Experience with programming languages such as Python, R, Scala, and Go.
- Advanced working SQL knowledge and experience with relational databases.
- Advanced working knowledge of NoSQL databases.
- Experience with message queuing, stream processing, and highly scalable big data stores.
- Experience manipulating, processing, and extracting value from large, disconnected datasets.
- Experience manipulating structured and unstructured data for analysis.
- Experience with data modeling tools and processes.
- Experience aggregating and transforming data from multiple datasets to create data products.
- Experience working in an Agile environment.
Nice To Haves
- Cloud database and data warehouse experience with AWS; Azure or GCP experience is mentioned as an alternative.
Responsibilities
- Lead and architect data pipelines and ingestion patterns to move raw data from data producers into an enterprise data ecosystem, with a focus on performance and reliability.
- Assess and understand ETL jobs, workflows, BI tools, and reports.
- Address technical inquiries concerning customization, integration, enterprise architecture, and data-product features and functionality.
- Develop database and data warehouse solutions in cloud environments, preferably AWS, or alternatively Azure or GCP.
- Inspect existing data pipelines, determine their purpose and functionality, and re-implement them efficiently in Databricks.
- Aggregate and transform data from multiple datasets to create data products.
- Support an Agile software development lifecycle.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
