Databricks Engineer

540 is seeking a Databricks Engineer for a short-term contract to enhance the efficiency of acquisition and procurement in the Department of Defense. The role involves designing, building, and maintaining data infrastructure using the Databricks platform, integrating data from various sources, and ensuring data quality and accuracy. The ideal candidate should have experience in Databricks, ETL functions, AWS cloud infrastructure, and proficiency in PySpark and Python programming. An active DoD Secret clearance is required.

• Design, build, implement, and maintain data ingestion, transformation, and processing infrastructure using the Databricks platform
• Integrate data from various source systems, transform, clean, and store it in designated target data repositories while ensuring data integrity and consistency
• Create, performance tune, monitor, and troubleshoot data pipelines to ensure data quality and accuracy
• Familiarity with AWS cloud infrastructure and functions
• Proficiency in the Data Warehouse ETL (Extract, Transform, Load) process
• Solid programming skills in PySpark, Python, and writing complex queries in Databricks
• Experience with consuming REST APIs
• Experience parsing XML files and JSON files
• Experience with Databricks framework (nice to have)

• Active DoD Secret clearance
• 5+ years professional experience working in Databricks
• 3+ years professional experience working in ETL functions and tools
• Familiarity with AWS cloud infrastructure / functions
• Proficiency in the Data Warehouse ETL (Extract, Transform, Load) process
• Solid programming skills in PySpark, Python, and writing complex queries in Databricks
• Experience with consuming REST APIs
• Experience parsing XML files and JSON files

• Familiarity with AWS cloud infrastructure / functions
• Proficiency in the Data Warehouse ETL (Extract, Transform, Load) process
• Solid programming skills in PySpark, Python, and writing complex queries in Databricks
• Experience with consuming REST APIs
• Experience parsing XML files and JSON files
• Experience with Databricks framework on Amazon Web Services (AWS) infrastructure (Nice to have)
• Familiarity with DoD procurement / acquisition processes (Nice to have)
• Knowledge of FPDS Atom Feed and USA-SPENDING/SAM REST APIs (Nice to have)
• Equal employment opportunity provided to all employees and applicants for employment without discrimination or harassment based on various characteristics protected by federal, state, or local laws.