Senior Databricks Platform Engineer (Remote US)

ICF•Reston, VA

2d•Remote

About The Position

ICF is seeking a Senior Databricks Platform Engineer to support an independent U.S. Government Agency. The role designs, builds, secures, and operates governed Azure Databricks Lakehouse capabilities for a cloud-based analytics platform, enterprise AI operations, and emerging data initiatives. The engineer develops reliable batch and streaming pipelines, administers Databricks workspaces and access, implements governance, and automates delivery through infrastructure as code and CI/CD. In collaboration with cloud architects, AI engineers, security analysts, Business Analysts, and mission owners, the role helps move data products from ingestion through validated, analysis-ready datasets and into production operations. The position is remote within the United States. The candidate must reside in the U.S., be authorized to work in the U.S., and perform all work in the U.S. U.S. citizenship is required due to federal contract requirements, and the candidate must satisfy FDIC personnel eligibility and background-investigation requirements. The position is based on full-time employment. The stated nationwide remote pay range is $98,614 to $202,850. Final pay considers relevant experience, skills, certifications, competencies, location, education, and contract provisions.

Requirements

BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.
More than 5 years of data engineering experience and at least 3 years specifically supporting Azure Databricks and related Azure data services.
Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala, with hands-on experience designing and optimizing large-scale ETL/ELT pipelines.
Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.
Experience configuring and managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization.
Experience implementing CI/CD or infrastructure as code for data platforms using Git and tools such as Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable technologies.
Experience with Agile delivery, data governance, data security, performance troubleshooting, and clear communication with technical and business stakeholders.
Ability to satisfy FDIC personnel eligibility and background-investigation requirements.
U.S. citizenship is required due to federal contract requirements.
Must reside in the U.S., be authorized to work in the U.S., and perform all work in the U.S.
Submit the application directly; applications submitted by third parties may be excluded from consideration.

Nice To Haves

Experience with Azure Databricks, ADLS Gen2, Unity Catalog, Delta Live Tables, Auto Loader, Azure Data Factory, and Azure Synapse.
Databricks Certified Data Engineer Associate or Professional certification, Databricks Generative AI Engineer certification, or comparable demonstrated experience.
Experience with streaming and real-time data using Kafka, Confluent, Structured Streaming, or similar technologies.
Experience with medallion architecture, CDC, dimensional modeling, Apache Iceberg, Snowflake, Oracle, SQL Server, or other enterprise data platforms.
Experience supporting Generative AI, NLP, RAG applications, vector stores, agents, or Databricks AI/ML workloads.
Experience with MLflow, model deployment pipelines, AWS SageMaker, AWS Bedrock, or cross-cloud data and AI architectures.
Experience implementing BCDR, high availability, disaster recovery, data retention, and enterprise security controls for cloud data platforms.
Experience supporting ATO processes, NIST or federal compliance controls, vulnerability remediation, SonarQube, Checkmarx, or Zero Trust implementation.
Experience with Docker, Kubernetes, REST APIs, FastAPI, Node.js, or cloud-native application integration.
Experience supporting federal data modernization programs and communicating technical decisions to agency stakeholders.
Ability to explain data platform architecture, trade-offs, risks, and operational impacts to technical and nontechnical stakeholders.
Ability to work effectively across data engineering, AI, cloud, security, application, and operations teams.
Ability to use structured problem solving to diagnose pipeline, performance, access, and deployment issues.
Ability to organize work across concurrent Agile priorities and maintain clear technical documentation and status communication.

Responsibilities

Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals for enterprise initiatives.
Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.
Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.
Configure Unity Catalog governance, including role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage.
Develop frameworks and interfaces that enable AI and analytics teams to access governed training, inference, and reporting data across CDMA, AlphaRex, and future DMS initiatives.
Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools.
Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services as required by the solution architecture.
Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and use Azure Monitor, Databricks logs, Splunk, or comparable tools for root cause analysis and operational reliability.
Support security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness activities for Databricks environments and data products.
Participate in Agile planning, code reviews, demos, and retrospectives; document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories.
Explain platform architecture, trade-offs, risks, and operational impacts to technical and nontechnical stakeholders; collaborate across data engineering, AI, cloud, security, application, and operations teams.
Organize work across concurrent Agile priorities and maintain clear technical documentation and status communication.