Senior Databricks Engineer

Koniag Government Services • Washington, DC

4d • Remote

About The Position

Koniag Emerging Technologies is seeking a highly qualified and self-motivated Senior Databricks Engineer to support a client in Washington, DC. This remote role requires an active Secret clearance. The engineer will design, develop, and deploy scalable data infrastructure and lead data engineering initiatives for the Department of State’s “4Ps” data lake, using the Databricks platform, modern data warehousing practices, and the Medallion Architecture.

The role combines hands-on pipeline engineering, architecture, data quality governance, API integration, automated deployments, data warehousing, performance tuning, dashboarding, and technical leadership. The engineer will collaborate with project managers, team members, and business owners to translate requirements into data solutions, support customer needs, and mentor other data engineering staff.

Requirements

• Active Secret clearance.
• Knowledge of object-oriented programming concepts and proficiency in one or more programming or scripting languages, such as Java, C#, Python, JavaScript, or PowerShell.
• At least 8 years of dedicated data engineering experience, including at least 3 years in a senior capacity focused on the Databricks ecosystem.
• Expert-level proficiency in PySpark and complex SQL.
• Ability to create, package, and manage custom Python libraries, including building Wheel files, for reuse across data pipelines and projects.
• Deep understanding of HTTP REST methods and hands-on experience extracting, paginating, and processing data from external APIs.
• Theoretical and practical understanding of modern data warehousing and hands-on experience building the Medallion Architecture.
• Hands-on experience automating Databricks deployments, including configuring and deploying Databricks Asset Bundles using Azure DevOps, creating YAML pipelines, managing service principals, and handling multi-workspace deployments.
• Strong familiarity with the cloud ecosystem surrounding Databricks, such as Azure Data Lake Storage Gen2 and Azure Key Vault.
• Proficiency with Git-based version control workflows.
• Concise and comprehensive written and oral communication skills for technical understanding and customer usage support.

Nice To Haves

• Experience with data products, BI tools, and visualization libraries such as Tableau, PowerBI, Jupyter Notebooks, and d3js, or analytics search tools such as Elasticsearch, SOLR, and Lucene.
• Experience working with PeopleSoft HR application data for reporting purposes.
• At least 3 years of .NET, Java, or similar object-oriented programming experience.
• Experience writing Oracle PL/SQL queries, packages, and procedures.
• Experience architecting Data Vault 2.0 solutions.

Responsibilities

• Collaborate with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs.
• Architect and implement scalable, highly available Databricks data pipelines using the Bronze, Silver, and Gold layers of the Medallion Architecture.
• Establish data quality standards and monitoring dashboards so business owners and responsible parties can track and address data issues.
• Develop clean, efficient, production-ready PySpark and SQL code for complex ETL/ELT processes, including DLT and structured streaming pipelines.
• Build data ingestion pipelines that retrieve data from external systems through REST API integrations.
• Implement infrastructure and pipelines as code, and automate deployments across Development, Staging, and Production using Databricks Asset Bundles and Azure DevOps.
• Develop, package, and maintain reusable custom Python libraries and modules for data projects and teams.
• Apply data warehousing concepts, including dimensional modeling and slowly changing dimensions, to optimize data storage and retrieval for analytics and business intelligence.
• Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs for efficiency and cost-effectiveness.
• Securely share data with external BI tools such as PowerBI or SAP BusinessObjects and develop dashboards and capabilities within Databricks.
• Serve as a technical subject-matter expert, conduct code reviews, establish best practices, and mentor other staff members.

Benefits

• Medical insurance
• Dental insurance
• Vision insurance
• 401(k) retirement plan
• Paid time off
• Paid parental leave
• Life insurance
• Disability insurance
• Flexible spending accounts
• Commuter benefits
• Tuition reimbursement

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed