Senior Databricks Engineer

Koniag Government Services, LLC•Washington, DC

1d•$150,000 - $165,000•Remote

About The Position

Koniag Emerging Technologies is seeking a highly qualified and self-motivated Senior Databricks Engineer to support a client located in Washington, DC. This remote role requires an active Secret clearance. The engineer will design, develop, and deploy scalable data infrastructure and lead data engineering initiatives using the Databricks platform, modern data warehousing practices, and the Medallion Architecture. The work includes building data pipelines supporting the Department of State’s “4Ps” data lake. The role also involves collaborating with project managers, team members, and business owners; establishing data quality standards and monitoring; developing production-ready pipelines and reusable Python libraries; automating deployments; optimizing Databricks and Spark performance; supporting data sharing and dashboarding; and providing technical leadership and mentorship.

Requirements

Active Secret clearance.
At least 8 years of dedicated data engineering experience, including at least 3 years in a senior capacity focused on the Databricks ecosystem.
Knowledge of object-oriented programming concepts and proficiency in one or more programming or scripting languages, such as Java, C#, Python, JavaScript, or PowerShell.
Expert-level proficiency in PySpark and complex SQL.
Proven ability to create, package, and manage custom Python libraries, including building Wheel files, for reuse across data pipelines and projects.
Deep understanding of HTTP REST methods, including GET, POST, and PUT, and hands-on experience extracting, paginating, and processing external API data.
Theoretical and practical understanding of modern data warehousing and hands-on experience implementing the Medallion Architecture.
Hands-on experience automating Databricks deployments and configuring and deploying Databricks Asset Bundles using Azure DevOps, including YAML pipelines, service principals, and multi-workspace deployments.
Strong familiarity with the cloud ecosystem surrounding Databricks, such as Azure Data Lake Storage Gen2 and Azure Key Vault.
Proficiency with Git-based version control workflows.
Concise and comprehensive written and oral communication skills for technical understanding and customer usage support.

Nice To Haves

Experience with data products, BI tools, and visualization libraries such as Tableau, PowerBI, Jupyter Notebooks, and d3js, and with analytics search tools such as Elasticsearch, SOLR, or Lucene.
Experience working with PeopleSoft HR application data for reporting.
At least 3 years of .NET, Java, or similar object-oriented programming experience.
Experience writing Oracle PL/SQL queries, packages, and procedures.
Experience architecting Data Vault 2.0 solutions.

Responsibilities

Collaborate with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs.
Architect and implement scalable, highly available Databricks data pipelines using the Bronze, Silver, and Gold layers of the Medallion Architecture.
Establish data quality standards and monitoring dashboards so business owners and responsible parties can track and address data issues.
Develop clean, efficient, production-ready PySpark and SQL code for complex ETL/ELT processes, including DLT and structured streaming pipelines.
Build robust ingestion pipelines using RESTful API integrations to retrieve data from third-party external systems.
Implement infrastructure and pipelines as code, using Databricks Asset Bundles and Azure DevOps to automate deployments across Development, Staging, and Production environments.
Develop, package, and maintain custom Python libraries and modules for reuse across data projects and teams.
Apply data warehousing concepts, including dimensional modeling and slowly changing dimensions, to optimize data storage and retrieval for analytics and business intelligence.
Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs for efficiency and cost-effectiveness.
Securely share data with external BI tools such as PowerBI and SAP BusinessObjects, and develop dashboards and capabilities within Databricks.
Serve as a technical subject matter expert, conduct code reviews, establish best practices, and mentor data engineering staff.