Principal Databricks Software Developer

PowerPlan • Atlanta, GA

16h • Hybrid

About The Position

PowerPlan is a profitable market leader with over 30 years of experience in asset-intensive financial systems and managing trillions of dollars of clients’ assets. PowerPlan is in a growth phase investing into next-generation solutions. We’re hiring a Principal Databricks Platform Engineer to take ownership of the health, reliability, performance, governance, and cost effectiveness of our Databricks platform. This is not a role for someone who only advises from the sidelines. You’ll be hands-on with notebooks, pipelines, jobs, clusters, Spark configurations, Delta Lake patterns, and Unity Catalog. You’ll be the person teams turn to when something technically works but is slower than it should be, more expensive than it needs to be, harder to operate than it should be, or fragile enough to become tomorrow’s production issue.

Requirements

• 8+ years in data engineering, platform engineering, or related software engineering roles, with at least 3+ years of deep hands-on Databricks experience.
• Production experience with Spark internals and performance tuning, including shuffle behavior, query planning, adaptive query execution, memory pressure, skew, spill, partitioning, and broadcast strategy.
• Strong working knowledge of Databricks compute options, cluster policies, job clusters, all-purpose clusters, instance pools, autoscaling, serverless eligibility, Photon, and workload right-sizing.
• Real experience reviewing and improving notebooks, jobs, and pipelines that other people built, including the ability to quickly identify the few design choices causing most of the runtime, cost, or reliability risk.
• Production experience with Delta Lake, Delta maintenance strategies, Structured Streaming, Auto Loader, materialized views or streaming tables, and data quality expectations.
• Strong understanding of Unity Catalog, workspace governance, lineage, permission models, catalog/schema organization, table documentation, and secure handling of sensitive data.
• Experience designing reliable job orchestration: task decomposition, dependency management, retries, repairs, timeouts, alerting, scheduling, max concurrent runs, and recoverability.
• A track record of identifying systemic reliability, performance, governance, or cost issues and driving practical remediation across teams.
• Strong communication skills, including the judgment to explain technical findings in a way that helps engineering leaders, product owners, and stakeholders make better decisions.
• A STEM degree, or equivalent depth earned through substantial production experience.

Nice To Haves

• Experience with Azure-based data platforms and enterprise cloud operations.
• Experience partnering with FinOps or platform teams to reduce cloud spend without sacrificing reliability or service-level expectations.
• Experience building reusable Databricks templates, platform automation, internal standards, or reference architectures.
• Familiarity with data privacy, security, auditability, and regulated business environments.
• Opinions, backed by evidence, about when Spark or Databricks is the right tool — and when it is not.
• Experience helping teams adopt AI-assisted engineering practices responsibly, especially for code review, test generation, documentation, and operational analysis.

Responsibilities

• Take ownership of the highest-risk Databricks workloads and platform patterns, from configuration review through remediation and production behavior.
• Review existing clusters, jobs, notebooks, pipelines, and workspace settings to identify cost waste, performance bottlenecks, reliability risks, security gaps, and governance drift.
• Set technical direction for Databricks usage across teams, including compute policies, Spark configuration standards, notebook design expectations, pipeline patterns, orchestration practices, and Unity Catalog governance.
• Raise the bar for Databricks code and architecture reviews by identifying anti-patterns such as oversized clusters, unnecessary shuffles, driver-side collections, unbounded full scans, brittle job dependencies, missing retry logic, poor Delta maintenance, and hardcoded environment assumptions.
• Partner with engineering teams to turn findings into practical improvements: better defaults, reusable templates, reference architectures, automation, monitoring, and clear standards that teams can actually follow.
• Serve as an escalation point for complex Databricks failures, performance regressions, pipeline instability, data quality issues, and cost anomalies.
• Translate technical risk into business impact so stakeholders understand why a slow job, missing lineage, weak guardrail, or inefficient cluster design matters.
• Make engineers around you better through pairing, reviews, design conversations, internal workshops, and examples that demonstrate what excellent Databricks engineering looks like.

Benefits

• This is a hybrid role that involves a combination of onsite work from our corporate office as well as work from home. While we strive to accommodate flexible working arrangements when sensible, there will be times when onsite work is required. This could include scheduled office days, team meetings, client meetings, or special events.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: High school or GED