Databricks Specialist

eTelligent Group LLC • Washington, DC

5h • Remote

About The Position

The IRS is modernizing its data platform to support analytics, reporting, and predictive modelling. We are seeking a skilled Databricks Specialist / Data Engineer to design, implement, and optimize big data pipelines and analytics solutions. The ideal candidate will have hands-on experience with Databricks, Apache Spark, and cloud-based data platforms, and a strong understanding of data governance, security, and compliance standards relevant to federal environments.

Requirements

• Hands-on experience with Databricks and Apache Spark in production environments.
• Strong data engineering skills: ETL/ELT pipelines, data modeling, and analytics.
• Proficient in Python, SQL, or Scala for data processing.
• Experience with cloud platforms such as AWS, Azure, or GCP, including cloud storage, compute, and security controls.
• Knowledge of Delta Lake, data lake architecture, and big data best practices.
• Experience working in federal or highly regulated environments preferred.
• IRS MBI clearance (MUST already be in place)

Nice To Haves

• Familiarity with data governance, metadata management, and lineage tracking.
• Experience integrating Databricks with BI tools like Power BI or Tableau.
• Knowledge of machine learning workflows in Databricks.
• Understanding of IRS modernization initiatives and taxpayer data sensitivity.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Work with large datasets from multiple sources, including structured and unstructured data, to support analytics and reporting.
• Collaborate with IRS teams to implement data lakehouse solutions and optimize data storage and retrieval.
• Develop and maintain Delta Lake tables and pipelines to ensure data reliability and ACID compliance.
• Implement data transformations, cleansing, and enrichment processes for downstream analytics and reporting.
• Work with Python, SQL, and other scripting languages to build and automate workflows.
• Ensure compliance with IRS security standards, NIST 800-53, and IRS 1075 for handling sensitive taxpayer data.
• Collaborate with data scientists, analysts, and other stakeholders to deliver actionable insights.
• Participate in performance tuning, troubleshooting, and optimization of Databricks clusters and workloads.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level