Databricks Engineer

540 is seeking a Databricks Engineer with an active DoD Secret clearance for a short-term contract. The engineer will be responsible for designing, building, implementing, and maintaining data ingestion, transformation, and processing infrastructure using the Databricks platform. They will also integrate data from various source systems, create and troubleshoot data pipelines, and ensure data quality and accuracy. The ideal candidate should have at least 5 years of professional experience in Databricks, proficiency in PySpark and Python, and familiarity with AWS cloud infrastructure.

• Design, build, implement, and maintain data ingestion, transformation, and processing infrastructure using the Databricks platform
• Integrate data from various source systems, transform, clean, and store it in designated target data repositories while ensuring data integrity and consistency
• Create, performance tune, monitor, and troubleshoot data pipelines to ensure data quality and accuracy
• Familiarity with AWS cloud infrastructure and functions
• Proficiency in the Data Warehouse ETL (Extract, Transform, Load) process
• Solid programming skills in PySpark, Python, and writing complex queries in Databricks
• Experience with consuming REST APIs
• Experience parsing XML files and JSON files

• Active DoD Secret clearance
• 5+ years professional experience working in Databricks
• 3+ years professional experience working in ETL functions and tools
• Familiarity with AWS cloud infrastructure/functions
• Proficiency in the Data Warehouse ETL process
• Solid programming skills in PySpark, Python, and writing complex queries in Databricks
• Experience with consuming REST APIs
• Experience parsing XML files and JSON files
• Nice to have: Experience with Databricks framework

• Familiarity with AWS cloud infrastructure / functions
• Proficiency in the Data Warehouse ETL (Extract, Transform, Load) process
• Solid programming skills in PySpark, Python, and writing complex queries in Databricks
• Experience with consuming REST APIs
• Experience parsing XML files and JSON files
• Experience with Databricks framework on Amazon Web Services (AWS) infrastructure
• Familiarity with DoD procurement / acquisition processes
• Knowledge of FPDS Atom Feed and USA-SPENDING/SAM REST APIs
• Equal employment opportunity provided to all employees and applicants for employment
• Prohibition of discrimination and harassment based on various characteristics protected by federal, state, or local laws.