15+)

InnoSoul•Austin, TX

5d•Hybrid

About The Position

The Databricks Engineer should design, develop, and optimize scalable data solutions on Databricks, leveraging PySpark or Scala for large-scale data processing. Build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting. Develop robust data models and implement data quality, validation, and governance frameworks. Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights. Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency. This role requires 8 or more years of experience, relies on experience and judgment to plan and accomplish goals, independently performs a variety of complicated tasks, and a wide degree of creativity and latitude is expected. The engineer will understand business objectives and problems, identify alternative solutions, and perform studies and cost/benefit analysis of alternatives. They will analyze user requirements, procedures, and problems to automate processing or to improve existing computer systems by conferring with personnel to analyze current operational procedures, identify problems, and learn specific input and output requirements. The engineer will write detailed descriptions of user needs, program functions, and steps required to develop or modify computer programs, and review computer system capabilities, specifications, and scheduling limitations to determine if requested programs or program changes are possible within existing systems.

Requirements

8+ years of experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
8+ years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
8+ years of experience in data warehousing and dimensional data modeling (star/snowflake schemas).
Proficiency in SQL and Python (or Scala) for large-scale data processing.
8+ years of experience designing and developing dashboards and applications natively within Databricks (e.g., Databricks SQL dashboards, Databricks Apps).
8+ years of experience implementing data governance, data quality, and data security practices.
8+ years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) for building and managing production data pipelines.
8+ years of experience with Delta Lake, medallion architecture (bronze/silver/gold layers), data lakehouse design, and creating and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools (e.g., Airflow).
Excellent communication skills, both verbal and written, including presenting insights to technical and business stakeholders.

Nice To Haves

Experience working in public sector or state government environments.
Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
Experience with CI/CD practices for data pipelines (DevOps, Git-based workflows).

Responsibilities

Design, develop, and optimize scalable data solutions on Databricks.
Leverage PySpark or Scala for large-scale data processing.
Build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting.
Develop robust data models.
Implement data quality, validation, and governance frameworks.
Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights.
Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.
Plan and accomplish goals, independently performing a variety of complicated tasks.
Understand business objectives and problems, identify alternative solutions, and perform studies and cost/benefit analysis of alternatives.
Analyze user requirements, procedures, and problems to automate processing or to improve existing computer systems.
Write detailed descriptions of user needs, program functions, and steps required to develop or modify computer programs.
Review computer system capabilities, specifications, and scheduling limitations to determine if requested programs or program changes are possible within existing systems.