Databricks Data Engineer

QED National • Austin, TX

65d

About The Position

Our public sector client in Austin, TX is seeking an experienced Databricks Data Engineer to join their data engineering and analytics team. This role focuses on designing, building, and optimizing scalable data pipelines and ETL/ELT workflows on the Databricks Unified Analytics Platform, integrated with Azure Data Factory. The ideal candidate is an expert in Apache Spark, Azure Data Lake, and Python, with deep experience in data governance, security, and automation using CI/CD. You will work closely with data scientists, analysts, and DevOps teams to deliver high-quality, production-ready data solutions that power analytics and machine learning initiatives.

Requirements

• 8+ years of experience in data engineering, including ETL/ELT development for structured and unstructured data.
• 5+ years of experience designing and optimizing data pipelines on Apache Spark / Databricks.
• Proven experience with Azure Data Factory, Azure Data Lake, and related cloud data services.
• Expertise in Python, R, and SQL for data processing and analysis.
• Hands-on experience with CI/CD automation, DevOps tools, and version control systems (e.g., Git).
• Strong background in data modeling, data governance, and data quality assurance.
• Experience with Unity Catalog or Delta Lake for data governance and security.
• Deep understanding of Apache Spark architecture, including RDDs, DataFrames, and Spark SQL.
• Skilled in troubleshooting complex data systems and improving pipeline performance.
• Experience working in Agile, multicultural environments.

Nice To Haves

• Experience with machine learning tools such as MLflow, Scikit-learn, or TensorFlow.
• Databricks Certified Associate Developer for Apache Spark.
• Microsoft Azure Data Engineer Associate certification.

Responsibilities

• Design and develop scalable data pipelines using Apache Spark on Databricks.
• Implement and maintain ETL/ELT workflows for both structured and unstructured data.
• Optimize Spark jobs for performance, scalability, and cost-efficiency.
• Integrate Databricks solutions with Azure Data Factory and other Azure cloud services.
• Design and manage data models, schemas, and database structures supporting analytics and operational workloads.
• Implement robust data validation, quality checks, and contribute to metadata management, data lineage, and cataloging.
• Apply data security and compliance best practices, including encryption, access control, and auditing.
• Automate deployment pipelines with CI/CD tools in collaboration with DevOps.
• Partner with data scientists, analysts, and stakeholders to align technical solutions with business goals.
• Troubleshoot and resolve performance bottlenecks and data integrity issues to ensure system reliability.

Benefits

• Competitive pay
• Comprehensive health, dental, and vision coverage
• 401(k) retirement plans
• Support of a dedicated team

What This Job Offers

• Industry: Administrative and Support Services
• Number of Employees: 101-250 employees