Spark / Databricks Platform Engineer

SMART TECH SKILLS LLC•Austin, TX

4d•Remote

About The Position

The Databricks Engineer designs, develops, and optimizes scalable data solutions on the Databricks platform, leveraging PySpark or Scala for large-scale data processing. Operating with a high degree of creativity and latitude, this role involves constructing ingestion pipelines, implementing medallion architecture (Bronze, Silver, Gold layers), and designing robust dimensional data models. The engineer collaborates with cross-functional stakeholders to gather requirements, perform cost/benefit analyses of alternative solutions, and implement data governance, security, and quality frameworks to support enterprise public health and business intelligence initiatives.

Requirements

8 or more years of experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
8 or more years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
8 or more years of experience in data warehousing and dimensional data modeling, including Star and Snowflake schemas.
8 or more years of professional proficiency utilizing SQL and Python (or Scala) for large-scale data processing.
8 or more years of experience designing and developing dashboards and applications natively within Databricks, such as Databricks SQL dashboards or Databricks Apps.
8 or more years of experience implementing data governance, data quality metrics, and data security practices.
8 or more years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT) to build and manage production pipelines.
8 or more years of experience with Delta Lake, medallion architecture (Bronze, Silver, Gold layers), data lakehouse design, and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools.
8 or more years of experience communicating technical specifications and presenting data-driven insights to technical and non-technical stakeholders.

Nice To Haves

1 or more years of experience working within public sector or state government environments.
1 or more years of experience implementing CI/CD practices for data pipelines, including DevOps and Git-based version control workflows.
Professional Databricks certification, such as Databricks Certified Data Engineer Associate or Professional.
Strong analytical, critical-thinking, and problem-solving skills with a high degree of creativity and latitude.
Excellent written and verbal communication skills to write detailed descriptions of user needs and document system capabilities.
Proactive, self-motivated approach to managing complex, cross-functional datasets and resolving workload issues.
Strong collaborative capability to work effectively with data engineers, database administrators, and business partners.

Responsibilities

Design, develop, and optimize highly scalable data solutions on Databricks using Apache Spark, PySpark, or Scala for large-scale data processing.
Build, configure, and maintain robust ETL/ELT ingestion pipelines utilizing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT).
Implement and maintain Delta Lake architectures and medallion database structures across Bronze, Silver, and Gold data layers.
Orchestrate, schedule, and monitor offline production jobs utilizing Lakeflow Jobs (formerly Databricks Workflows) or equivalent tools such as Apache Airflow.
Design, develop, and support comprehensive enterprise data warehouses utilizing dimensional data modeling, including Star and Snowflake schemas.
Implement rigorous data quality validation, data security controls, platform access models, and data governance frameworks.
Analyze system specifications, evaluate operational limitations, and perform data audits to ensure scalability, security, and cost efficiency.
Collaborate directly with business stakeholders, program managers, and technical peers to understand operational objectives, identify problems, and analyze current procedures.
Translate high-level business goals into formal technical requirements, system design documentation, and cost/benefit analyses.
Develop dynamic analytical dashboards and reporting solutions natively within Databricks, including Databricks SQL dashboards and Databricks Apps, to deliver actionable operational insights.