Senior Databricks Data Engineer
About The Position
KIBO is a registered United States company providing inspection, analysis, certification, and compliance services to governments worldwide. The Senior Databricks Data Engineer will lead the design, development, and optimization of data pipelines and data platforms in a Databricks-on-AWS environment. Working closely with the Principal AI Architect, the role will build scalable, secure, and governed data solutions supporting analytics, reporting, and AI initiatives. The position involves configuring and optimizing Databricks workspaces and supporting AWS infrastructure; designing batch and streaming pipelines; implementing Medallion Architecture and analytics-ready data models; and establishing data governance and security controls. The engineer will integrate data from APIs, file transfers, and operational systems, implement Git-based development and CI/CD practices, and collaborate with cross-functional teams in an Agile environment. In the first few months, the role is expected to establish and secure the Databricks environment and governance framework, deliver initial ingestion pipelines and curated Bronze, Silver, and Gold data layers, implement CI/CD and data-quality monitoring standards, and support a transition toward near-real-time incremental ingestion. The position reports to the Principal AI Architect and works closely with data, AI, analytics, and engineering stakeholders.
Requirements
- At least 5 years of experience in Data Engineering, including at least 2 years of hands-on Databricks experience in production.
- Strong experience with the Databricks Lakehouse Platform, Delta Lake, Unity Catalog, and PySpark.
- Advanced SQL and Python development skills.
- Experience implementing Medallion Architecture and data models for analytics and reporting.
- Experience with Delta Live Tables, change data capture (CDC), incremental processing, and data-quality controls.
- Working knowledge of AWS services, including S3, IAM, storage credentials, and external locations.
- Experience with Git, CI/CD pipelines, and automated testing practices.
- Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent experience.
Nice To Haves
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience with Spark performance tuning and cost optimization.
- Software engineering background with API development experience.
- Experience developing and testing locally using PySpark, Delta Lake, and pytest.
- Experience supporting AI/ML, financial, trade, customs, or other regulated-data environments.
Responsibilities
- Configure, manage, and optimize Databricks workspaces and supporting AWS infrastructure.
- Design and build scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Auto Loader.
- Implement and maintain Medallion Architecture (Bronze, Silver, Gold) and analytics-ready data models.
- Configure Unity Catalog, permissions, row-level security, column masking, and data-access controls.
- Ingest data from third-party APIs, file transfers, and operational systems using modern Databricks patterns.
- Implement Git-based development workflows, automated testing, and deployment processes across environments.
- Collaborate with cross-functional teams in an Agile environment to deliver reliable, high-quality data solutions.
- Establish and secure the Databricks environment and governance framework.
- Deliver initial ingestion pipelines and curated Bronze, Silver, and Gold data layers.
- Implement CI/CD processes and data-quality monitoring standards.
- Support the transition from batch processing to near-real-time incremental data ingestion.
Benefits
- Competitive compensation and comprehensive benefits.
- Opportunity to build a modern Databricks data platform from the ground up.
- Exposure to cutting-edge data engineering and AI initiatives.
- A collaborative, mission-driven environment with significant ownership and impact.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
