Senior Data Engineer - Databricks
About The Position
Steampunk is seeking a seasoned Senior Data Engineer to work with its team and clients to develop enterprise-grade data platforms, services, and pipelines in Databricks. The role calls for a technologist with strong communication and customer-service skills and an interest in data and problem solving. The engineer will contribute to the growth of Steampunk’s AI & Data Exploitation Practice. The position involves leading and architecting data migrations using Databricks, with an emphasis on performance, reliability, and scalability. The engineer will assess existing ETL jobs, workflows, data marts, BI tools, and reports; respond to technical questions about customization, integration, enterprise architecture, and data-product functionality; and support an Agile software development lifecycle. The role also involves working with cloud database, data warehouse, and data mart solutions, preferably on AWS, or alternatively Azure or GCP. Steampunk states that compensation depends on factors including geographic location, contractual requirements, education, knowledge, skills, competencies, and experience. The projected annual compensation range is $125,000 to $165,000. Applicants are expected to be on camera during interviews and assessments; Steampunk may take a picture to verify identity and prevent fraud.
Requirements
- Ability to hold a position of public trust with the US government.
- 5–7 years of industry experience coding commercial software.
- 5–7 years of direct Data Engineering experience.
- Experience with Databricks, Apache Spark, Delta Lake, or similar big-data tools.
- Experience with relational SQL, preferably T-SQL; alternatively pgSQL or MySQL.
- Experience with data-pipeline and workflow-management tools such as Databricks Workflows, Airflow, or Step Functions.
- Experience with AWS cloud services, including Databricks on AWS, S3, EC2, or RDS; Azure equivalents are also noted.
- Experience with object-oriented or object-function scripting languages such as PySpark/Python, Java, C++, or Scala.
- Experience with Data Lakehouse architecture and Delta Lake or Apache Iceberg.
- Advanced SQL knowledge and experience with relational databases, query authoring, and optimization, plus working familiarity with a variety of databases.
- Experience manipulating, processing, and extracting value from large, disconnected datasets.
- Experience manipulating structured and unstructured data.
- Experience architecting transactional and warehouse data systems.
- Experience with the SDLC, CI/CD, and operating in dev/test/prod environments.
- Commitment to data governance.
- Experience working in an Agile environment.
- Key skill sets include Databricks, SQL, PySpark/Python, and AWS.
Nice To Haves
- Experience with data-cataloging tools such as Informatica EDC, Unity Catalog, Collibra, Alation, Purview, or DataZone.
- Experience with cloud data solutions on AWS is preferred; Azure and GCP are listed as alternatives.
- Experience with T-SQL is preferred; pgSQL and MySQL are listed as alternatives.
Responsibilities
- Lead and architect data migrations using Databricks, focusing on performance, reliability, and scalability.
- Assess and understand ETL jobs, workflows, data marts, BI tools, and reports.
- Address technical inquiries concerning customization, integration, enterprise architecture, and the features and functionality of data products.
- Work with cloud database, data warehouse, and data mart solutions.
- Support an Agile software development lifecycle.
- Inspect existing data pipelines, determine their purpose and functionality, and re-implement them efficiently in Databricks.
- Support project teams of developers and data scientists building web-based interfaces, dashboards, reports, and analytics or machine-learning models.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
