Databricks Engineer

Perficient•Town and Country, MO

1h•$45,864 - $101,520

About The Position

The Databricks Data Engineer drives the design, automation, and scaling of a cloud-native data platform. Working closely with engineering and business leaders, this role designs and builds data integration solutions that support enterprise analytics and decision-making. The engineer uses cloud technologies, CI/CD pipelines, and Power BI to deliver scalable, high-performance data solutions, enabling self-service analytics and supporting data accessibility, governance, and integrity. The role is part of an agile team that participates in Scrum ceremonies and delivers continuous value to stakeholders. The engineer collaborates with a small data engineering team to evolve the data platform, provides technical leadership and mentoring, and contributes to planning, requirements gathering, design, development, testing, deployment, troubleshooting, documentation, and training. The work includes building batch and streaming pipelines with Databricks, integrating with Azure or AWS services, and developing analytical models for BI and AI use cases.

Requirements

At least 5 years of data engineering experience, including at least 3 years in a lead or senior data engineer role.
At least 3 years of hands-on Databricks experience using Python, SQL, PySpark, Notebook Repositories with medallion architecture L2, L3, and L4, and CI/CD.
At least 3 years of Databricks experience developing large-scale data processing, storage, or distribution systems.
Experience with an AWS and/or Azure cloud environment.
Experience with Notebook Repositories using medallion architecture L2, L3, and L4, as well as unit testing frameworks.
DevOps experience with Databricks CI/CD, including Databricks Asset Bundles, and Git.
At least 3 years of experience working in a large multi-vendor environment with multiple project teams.
At least 5 years of experience working in a complex Big Data/Lakehouse environment.
At least 5 years of experience with JIRA, GitHub, Git, and other code management toolsets.
Ability to use AI tools to enhance productivity, streamline workflows, and support data-informed task execution.
Understanding of AI capabilities and limitations, including ethical considerations.
Flexible, adaptable approach and discipline to manage multiple responsibilities and varied environments.
Ability to produce high-quality products within deadlines.
Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field.

Nice To Haves

Experience with ELT/ETL development, patterns, and tooling.
Experience with SQL, including Postgres and RDBMS platforms.
Experience with various IDEs, code repositories, and unit testing frameworks.
Familiarity with AI-enhanced platforms.
Certification in Databricks, Spark, AWS, Azure, or other cloud platforms.
Experience with ETL/ELT data platforms such as Informatica, DataStage, Fivetran, Matillion, and dbt.
Familiarity with ML capabilities such as MLflow, TensorFlow, or Scikit-learn.
Master’s degree in Computer Science or a related field.
Client-facing or consulting experience.

Responsibilities

Collaborate with a small team of data engineers to evolve the data platform and promote innovation, accountability, and continuous improvement.
Work with the lead on technical planning and requirements gathering.
Provide technical leadership, mentoring, and support for development and continual service improvement.
Build effective working relationships and adapt to working with team members at various experience levels.
Define and develop data ingestion, validation, transformation, and data engineering code with the data engineering team.
Lead the design, development, testing, and deployment of scalable batch and streaming data pipelines using Databricks.
Build data jobs for ingestion, cleansing, transformation, aggregation, governance, and sharing.
Define and enforce Databricks best practices for notebooks, job orchestration, metadata-driven frameworks, data quality checks, and Spark job optimization.
Develop Databricks platform components using Python, SQL, PySpark, Notebook Repositories with medallion architecture L2, L3, and L4, and CI/CD.
Integrate Databricks solutions with Azure services such as ADF, ADLS, Event Hub, and Azure DevOps, or AWS services such as S3, Lambda, and Glue.
Design and implement analytical data models and star schemas for BI and AI use.
Collaborate with data scientists and analysts to meet data requirements.
Implement logging, monitoring, and exception handling for data workflows.
Participate in code reviews and CI/CD pipeline development.
Document code artifacts and contribute to user documentation and runbooks.
Troubleshoot deployments across environments and provide test support.
Participate in design, demo, prototype, testing, and training sessions with business users and IT associates.