Systems Analyst 3 – Databricks Data Engineer
About The Position
The Texas Health and Human Services Commission (HHSC) is seeking an experienced Systems Analyst 3 – Databricks Data Engineer to design, develop, and optimize enterprise-scale data engineering solutions using the Databricks Lakehouse Platform. This role will be responsible for building scalable ETL/ELT pipelines, implementing modern data architectures, and delivering high-quality analytical solutions that support agency-wide reporting and decision-making. The ideal candidate will have extensive experience with Databricks, Apache Spark, SQL, Python/Scala, Delta Lake, Medallion Architecture, and Lakeflow Declarative Pipelines (formerly Delta Live Tables). This position requires strong technical expertise, excellent communication skills, and the ability to collaborate with business and technical stakeholders.
Requirements
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
- 8+ years of IT experience supporting enterprise technology solutions.
- 8+ years of hands-on experience with Databricks and Apache Spark.
- Extensive experience developing ETL/ELT pipelines in Databricks.
- Strong expertise in SQL and Python (or Scala).
- Experience with Delta Lake and Databricks Lakehouse architecture.
- Experience implementing Lakeflow Declarative Pipelines (DLT).
- Experience designing dimensional data models (Star/Snowflake).
- Strong knowledge of Medallion Architecture (Bronze, Silver, Gold).
- Experience creating Databricks SQL Dashboards and Databricks Apps.
- Experience implementing enterprise data governance, security, and data quality standards.
- Experience scheduling data workflows using Lakeflow Jobs or Airflow.
- Excellent verbal and written communication skills.
- Experience working in Texas State Government or other public sector organizations.
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience implementing CI/CD pipelines for data engineering projects.
- Experience with Git-based version control and DevOps best practices.
Responsibilities
- Design, develop, and optimize scalable data solutions using the Databricks Lakehouse Platform.
- Build and maintain high-performance ETL/ELT data pipelines using Apache Spark (PySpark or Scala).
- Develop and manage Lakeflow Declarative Pipelines (formerly Delta Live Tables) for enterprise data processing.
- Design and implement Medallion Architecture (Bronze, Silver, Gold layers) for data transformation.
- Create efficient dimensional data models including star and snowflake schemas.
- Develop Databricks SQL dashboards and Databricks Apps to provide actionable business insights.
- Implement robust data governance, quality validation, and security frameworks.
- Optimize data processing workloads for performance, scalability, and cost efficiency.
- Schedule and monitor production workloads using Lakeflow Jobs or comparable orchestration tools such as Airflow.
- Collaborate with business users, data architects, analysts, and developers to gather requirements and deliver solutions.
- Troubleshoot production issues and continuously improve platform performance.
- Present technical findings and recommendations to both technical and non-technical stakeholders.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
