Data Engineer - Databricks

Steampunk • McLean, VA

1d • $110,000 - $160,000

About The Position

Steampunk is seeking a seasoned Data Engineer to work with its team and clients to develop enterprise-grade data platforms, services, and pipelines in Databricks. The role calls for a technologist with strong communication and customer-service skills and a passion for data and problem solving.

The Data Engineer will lead and architect data migrations using Databricks, with a focus on performance, reliability, and scalability. The role includes assessing existing ETL jobs, workflows, data marts, BI tools, and reports; addressing technical inquiries about customization, integration, enterprise architecture, and data-product functionality; and supporting an Agile software development lifecycle. The engineer will contribute to the growth of Steampunk’s AI & Data Exploitation Practice.

The position requires the ability to hold a position of public trust with the U.S. government. The projected annual compensation range is $110,000 to $160,000. Applicants are expected to be on camera during interviews and assessments; Steampunk may take a picture to verify identity and prevent fraud.

Requirements

• Ability to hold a position of public trust with the U.S. government.
• Bachelor’s degree and 5 years of experience, or an equivalent combination of education and experience.
• 2–4 years of industry experience coding commercial software.
• 2–4 years of direct Data Engineering experience.
• Experience with Databricks, Apache Spark, Delta Lake, or similar big-data tools.
• Advanced working knowledge of SQL and experience with relational databases, query authoring, and query optimization; T-SQL is preferred, with pgSQL or MySQL as alternatives.
• Experience with data pipeline and workflow management tools such as Databricks Workflows, Airflow, or Step Functions.
• Experience with AWS cloud services, including Databricks on AWS, S3, EC2, and RDS, or Azure equivalents.
• Experience with object-oriented or object-function scripting languages such as PySpark/Python, Java, C++, or Scala.
• Experience with Data Lakehouse architecture and Delta Lake or Apache Iceberg.
• Experience manipulating, processing, and extracting value from large, disconnected datasets.
• Experience manipulating structured and unstructured data.
• Experience architecting transactional and warehouse data systems.
• Experience with the SDLC, CI/CD, and operating in development, test, and production environments.
• Commitment to data governance.
• Experience working in an Agile environment.
• Strong communication and customer-service skills and a passion for data and problem solving.

Nice To Haves

• Experience with data cataloging tools such as Informatica EDC, Unity Catalog, Collibra, Alation, Purview, or DataZone.
• Experience with database, data warehouse, or data mart solutions in the cloud; AWS is preferred, with Azure or GCP as alternatives.
• Experience with T-SQL; pgSQL or MySQL are alternatives.

Responsibilities

• Lead and architect data migrations using Databricks, focusing on performance, reliability, and scalability.
• Assess and understand ETL jobs, workflows, data marts, BI tools, and reports.
• Address technical inquiries concerning customization, integration, enterprise architecture, and data-product features and functionality.
• Support an Agile software development lifecycle.
• Contribute to the growth of the AI & Data Exploitation Practice.
• Inspect existing data pipelines, determine their purpose and functionality, and re-implement them efficiently in Databricks.
• Support project teams of developers and data scientists building web-based interfaces, dashboards, reports, and analytics or machine-learning models.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior