Senior Databricks Engineer

Worldlink•Frisco, TX

8d•Onsite

About The Position

WorldLink is seeking a hands-on Senior Databricks Engineer to support a greenfield Databricks implementation. The engineer will design, develop, test, and deploy scalable batch and real-time data engineering solutions, while reverse engineering existing AWS-based processing to identify the transformations, business rules, dependencies, and integration requirements that must be preserved or redesigned for the target architecture. The role will deliver prioritized capabilities and help establish reusable engineering patterns, development standards, governance, deployment practices, and best practices that improve platform scalability, reliability, performance, and maintainability. It includes building and operating Databricks pipelines and medallion data layers; developing ingestion, streaming, transformation, and downstream integration solutions; implementing Unity Catalog governance; optimizing and troubleshooting production workloads; and validating parity between legacy AWS processing and Databricks pipelines. The engineer will also contribute to technical design and reviews, documentation, testing, CI/CD and deployment practices, and operational readiness. The role involves working with client teams, providing hands-on knowledge transfer and mentoring, and recommending automation and engineering improvements. The position is full-time, W2, and onsite in Frisco, Texas. Some additional work beyond the normal schedule and periodic travel may be required.

Requirements

Bachelor’s degree in computer science, information systems, engineering, or a related field; equivalent experience may be considered.
At least 6 years of data engineering or software engineering experience, including significant experience designing, developing, and supporting enterprise-scale data platforms.
At least 3 years of hands-on Databricks experience developing and operating production data engineering solutions.
Production experience with Databricks Auto Loader, including incremental cloud object storage ingestion, schema inference and evolution, schema hints, rescued data, checkpoint/state management, backfill and reprocessing, and scalable file discovery.
Production experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables), including batch and streaming pipelines, data quality expectations, quarantine handling, streaming tables and materialized views, refresh modes, dependency management, monitoring, and alerting.
Production experience implementing Unity Catalog, including catalog, schema, and volume design; external locations; storage credentials; grants; row- and column-level access controls; lineage; and promotion across environments.
Hands-on Delta Lake experience, including MERGE patterns, Change Data Feed, time travel, OPTIMIZE, Z ORDER and/or liquid clustering, VACUUM and retention policies, and large-scale partitioning strategies.
Hands-on Apache Spark, PySpark, and Spark SQL experience, including complex transformations and production performance tuning.
Ability to diagnose and optimize Spark workloads using partition sizing, shuffle optimization, join strategy, skew handling, and Spark UI analysis.
Production Structured Streaming experience, including watermarking, late-arriving data, stateful processing, checkpointing, and exactly-once processing considerations.
Experience designing and implementing Medallion Architecture, including personally building Bronze, Silver, and Gold layers and defining layer responsibilities and boundaries.
Experience with Databricks Workflows and deployment as code, including job dependencies, retry and failure handling, alerting, and Git-based environment promotion using Databricks Asset Bundles, Terraform, or comparable automation.
Strong understanding of distributed data processing, data optimization, scalability, and production pipeline resiliency.
Experience implementing production-grade data quality, error handling, monitoring, logging, observability, and pipeline recovery.
Experience with CI/CD, automated deployment practices, Git source control, branching, pull requests, and peer code reviews.
Hands-on AWS data engineering experience sufficient to analyze existing implementations and translate required functionality into a Databricks architecture.
Strong hands-on experience developing and debugging AWS Glue ETL jobs with PySpark and Python, including DynamicFrames versus DataFrames, job bookmarks, connections, parameters, worker sizing, and runtime/version considerations.
Ability to analyze unfamiliar and undocumented AWS Glue code and identify transformations, data movement, dependencies, and business rules.
Strong working knowledge of Amazon S3 bucket and prefix structures, partitioning, formats, compression, access patterns, and downstream ingestion considerations.
Experience with Amazon Redshift data structures, distribution and sort strategies, COPY/UNLOAD patterns, stored procedures, and identifying embedded transformation or business logic.
Strong understanding of AWS IAM and data access patterns, including roles, assumed-role relationships, policies, and authentication mechanisms, sufficient to trace pipeline access and translate requirements into Databricks and Unity Catalog access patterns.
Working knowledge of AWS Glue Crawlers and Glue Data Catalog, including schema inference, schema drift, partition management, and mapping catalog definitions to Unity Catalog.
Working knowledge of Redshift Spectrum and external S3-backed data access patterns.
Working knowledge of supporting AWS data, orchestration, monitoring, and security services such as Lambda, Step Functions, EventBridge, Athena, CloudWatch, and Secrets Manager.
Ability to trace end-to-end AWS data pipelines across services, identifying sources, transformations, orchestration, dependencies, security and access requirements, failure handling, and downstream consumers.
Experience reverse engineering undocumented legacy data pipelines where business rules and processing requirements are embedded in application, ETL, orchestration, or database code.
Experience identifying pipeline behavior and dependencies when original developers or complete technical documentation are unavailable.
Experience reconciling data and validating parity between legacy and rebuilt pipelines, investigating discrepancies, and confirming equivalent business outcomes.
Ability to distinguish business logic that must be preserved from legacy platform workarounds or constraints and redesign appropriately for Databricks.
Ability to troubleshoot complex data engineering and production issues independently while collaborating with an integrated delivery team.
Experience working in Agile delivery environments and delivering against prioritized product backlogs.
Self-motivated and able to work in a team-based or independent environment.
Detail-oriented with strong organization skills.
Ability to work in a fast-paced environment with limited supervision and exercise discretion.
Strong communication and collaboration skills, including the ability to work across engineering, architecture, product, and business teams.

Responsibilities

Design, develop, test, and deploy scalable Databricks data pipelines and transformation workflows for a greenfield implementation.
Reverse engineer existing AWS data processing, including undocumented code and pipeline logic, to identify transformations, business rules, dependencies, orchestration, and integration requirements.
Build, enhance, and maintain Bronze, Silver, and Gold data layers for customer data models and downstream consumption.
Design and implement Databricks Auto Loader ingestion, including schema management, checkpointing, incremental processing, backfills, reprocessing, and file ingestion patterns.
Develop and operate Lakeflow Declarative Pipelines for batch and streaming workloads, including data quality expectations, error handling, dependencies, monitoring, and recovery.
Develop batch and real-time data processing solutions using Databricks, Structured Streaming, Apache Spark, PySpark, Spark SQL, and Delta Lake.
Design and optimize Delta Lake solutions using MERGE patterns, Change Data Feed, partitioning and clustering, retention, and optimization strategies.
Implement Unity Catalog governance structures, access controls, lineage, and promotion across development, test, and production environments.
Develop and support integrations between Databricks, Adobe, AWS services, and other downstream systems.
Analyze AWS Glue, Amazon S3, Amazon Redshift, Redshift Spectrum, and supporting AWS services to determine existing behavior, business rules, dependencies, security requirements, and target-state implementation needs.
Establish reusable Databricks Workflows, deployment-as-code patterns, frameworks, utilities, and engineering standards.
Apply engineering best practices for data quality, performance, scalability, observability, reliability, security, and maintainability.
Tune Spark workloads, pipelines, queries, streaming processes, and data structures, including partitioning, shuffle behavior, join strategies, data skew, and execution characteristics.
Troubleshoot pipeline, integration, data quality, performance, and production issues; participate in root cause analysis and remediation.
Develop automated validation and testing approaches for data product accuracy, completeness, and reliability.
Perform legacy-to-target parity validation, reconcile discrepancies, and confirm preservation of required business rules and data outcomes.
Distinguish required business functionality from legacy technical workarounds and redesign appropriately for the target architecture.
Participate in design discussions, peer code reviews, technical reviews, and solution refinement.
Work within client development, CI/CD, security, governance, and deployment processes.
Produce and maintain technical documentation, including data flows, implementation details, development patterns, operational procedures, reverse-engineering findings, and troubleshooting guidance.
Provide knowledge transfer and mentoring to client engineers to support internal Databricks capabilities and long-term platform ownership.
Use approved AI-assisted development tools where appropriate to improve development velocity, code quality, testing, documentation, and engineering efficiency.
Identify and recommend automation and engineering improvements that increase delivery speed, solution quality, operational reliability, and production recovery.