Databricks Engineer

TATA Consulting Services-posted 30 days ago

$100,000 - $120,000/Yr

Full-time • Mid Level

O Fallon, MO

5,001-10,000 employees

Professional, Scientific, and Technical Services

Resume

Match Score

Upload and Match ResumeTrack Jobs with Teal

Databricks Engineer Job Title : Databricks Engineer Experience Required - 8+ Years Must Have Technical/Functional Skills Tenured in the fields of Computer Science/Engineering or Software Engineering. Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery. Deep hands-on experience with Databricks and Python. Spark and SQL Expertise: Writing performant Spark SQL and PySpark transformations. Optimizing joins, window functions, and aggregations. Handling large-scale data with partitioning and caching strategies Experience with real-time data processing and streaming pipelines. Proficient in both Delta Live Tables (DLT) and Workflows - Understanding of Lakeflow Declarative Pipelines is an asset. DevOps and CI/CD - Using Databricks Asset Bundles (DABs) for deployment - Integrating with Git, Bitbucket, and Jenkins for version control and automation Proven track record using dbt for building robust, testable data transformation workflows following TDD. Strong grasp of cloud-based database infrastructure (AWS, Azure, or GCP). Skilled in developing insightful dashboards and scalable data models using Power BI. Expert in SQL development and performance optimization. Demonstrated success in building and maintaining data observability tools and frameworks. Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations.

Focus on standing up/adopting tools that will enable both internal and external customers.
Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
Work efficiently within a high security, PII and PCI-DSS environments
Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
Support architects and engineers as they design and build effective, agile applications and platforms
Design, develop, and optimize batch and real-time data pipelines using Medallion Architecture, preferably on Databricks cloud data platforms
Build data transformation workflows using dbt, with a strong focus on Test-Driven Development (TDD) and modular design.
Implement and manage CI/CD pipelines using GitLab and Jenkins, enabling automated testing, deployment, and monitoring of data workflows.
Drive DataOps practices by integrating testing, monitoring, versioning, and collaboration into every phase of the data pipeline lifecycle.
Build scalable and reusable data models that support business analytics and dashboarding in Power BI.
Develop and support real-time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming) for near-instant data availability.
Establish and implement data observability practices, including monitoring data quality, freshness, lineage, and anomaly detection across the platform.
Plan and own deployments, migrations, and upgrades across data platforms and pipelines to minimize service impacts, including developing and executing mitigation plans.
Collaborate with stakeholders to understand data requirements and deliver reliable, high-impact data solutions.
Document pipeline architecture, processes, and standards, promoting consistency and transparency across the team.
Apply exceptional problem-solving and analytical skills to troubleshoot complex data and system issues.
Demonstrate excellent written and verbal communication skills when collaborating across technical and non-technical teams.

Tenured in the fields of Computer Science/Engineering or Software Engineering.
Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery.
Deep hands-on experience with Databricks and Python.
Spark and SQL Expertise: Writing performant Spark SQL and PySpark transformations. Optimizing joins, window functions, and aggregations. Handling large-scale data with partitioning and caching strategies
Experience with real-time data processing and streaming pipelines.
Proficient in both Delta Live Tables (DLT) and Workflows
DevOps and CI/CD - Using Databricks Asset Bundles (DABs) for deployment - Integrating with Git, Bitbucket, and Jenkins for version control and automation
Proven track record using dbt for building robust, testable data transformation workflows following TDD.
Strong grasp of cloud-based database infrastructure (AWS, Azure, or GCP).
Skilled in developing insightful dashboards and scalable data models using Power BI.
Expert in SQL development and performance optimization.
Demonstrated success in building and maintaining data observability tools and frameworks.
Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations.

Understanding of Lakeflow Declarative Pipelines is an asset.

Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Track Jobs with Teal

Job Search Resources

•

AI Resume Builder

•

Databricks Resume Examples

•

Databricks Cover Letter Examples

Databricks Engineer

Job Search Resources

Tools

Career Hubs

Guides

Company