Databricks Architect/ Senior Data Engineer
About The Position
The Databricks Senior Data Engineer will play a critical role in designing, implementing, and optimizing enterprise-scale data platforms using the Databricks Lakehouse architecture. This role combines architecture leadership with hands-on engineering, focusing on building scalable, secure, and high-performance data pipelines and platforms. The ideal candidate will establish coding standards, define data architecture frameworks such as the Medallion Architecture, and guide the end-to-end development lifecycle of modern data solutions. This individual will collaborate with cross-functional stakeholders including data engineers, BI developers, analysts, and business leaders to deliver robust data platforms that enable advanced analytics, reporting, and AI-driven decision-making.
Requirements
- 8+ years of experience in data engineering, with strong hands-on expertise in Databricks and Apache Spark.
- Proven experience designing and implementing scalable ETL/ELT pipelines in cloud environments.
- Strong programming skills in Python and SQL; experience with PySpark required.
- Hands-on experience with Databricks Lakehouse, Delta Lake, and distributed data processing.
- Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP (Azure preferred).
- Experience with CI/CD pipelines, Git, and DevOps practices for data engineering.
- Strong understanding of data architecture, data modeling, and performance optimization.
- Experience working with cross-functional teams to deliver enterprise data solutions.
- Tackles complex data challenges, ensuring data quality and reliable delivery.
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
- Experience designing enterprise-scale data platforms and modern data architectures.
Nice To Haves
- Experience in GenAI application development
- Experience with data integration tools such as Azure Data Factory or similar platforms.
- Familiarity with cloud data warehouses such as Databricks, Snowflake, or Azure Fabric.
- Experience supporting analytics, reporting, or AI/ML workloads is highly desirable.
- Databricks, Azure, or cloud certifications are preferred.
- Strong problem-solving, communication, and technical leadership skills.
- Technical Proficiency in: Databricks, Apache Spark, PySpark, Delta Lake Python, SQL, Scala (preferred) Cloud platforms: Azure (preferred), AWS, or GCP Azure Data Factory, Kafka, and modern data integration tools Data warehousing: Databricks, Snowflake, or Azure Fabric DevOps tools: Git, Azure DevOps, CI/CD pipelines Data architecture, ETL/ELT design, and performance optimization
Responsibilities
- Architect and design scalable, reliable data platforms and complex ETL/ELT and streaming workflows for the Databricks Lakehouse Platform (Delta Lake, Spark).
- Write, test, and optimize code in Python, PySpark, and SQL for data ingestion, transformation, and processing.
- Implement CI/CD, monitoring, and automation (e.g., with Azure DevOps, DBX) for data pipelines.
- Work with BI developers, analysts, and business users to define requirements and deliver data-driven solutions.
- Tune delta tables, Spark jobs, and SQL queries for maximum efficiency and scalability.
Benefits
- OZ is committed to creating a continuum between work and life by allowing people to work remotely.
- We offer competitive compensation and a comprehensive benefits package.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Number of Employees
1-10 employees
