Senior Databricks Engineer

Worldlink•Frisco, TX

7d•Onsite

About The Position

WorldLink is seeking a hands-on Senior Databricks Engineer to support a greenfield Databricks implementation. The engineer will design, develop, test, and deploy scalable batch and real-time data engineering solutions, while reverse engineering existing AWS-based processing to identify transformations, business rules, dependencies, and integration requirements that need to be preserved or redesigned in the target architecture. The role will build and maintain Bronze, Silver, and Gold data layers; implement ingestion, streaming, governance, and deployment solutions; and establish reusable engineering patterns, development standards, and best practices. The position also involves performance optimization, production troubleshooting, legacy-to-target parity validation, technical documentation, knowledge transfer, mentoring, and identifying opportunities to improve automation and engineering efficiency. The role is based onsite in Frisco, Texas, and may require periodic travel and additional time beyond the normal schedule. WorldLink describes its culture as collaborative, team-oriented, ambitious, experimental, and focused on learning and career advancement.

Requirements

Bachelor’s degree in computer science, information systems, engineering, or a related field; equivalent experience may be considered.
At least 6 years of data engineering or software engineering experience, including significant experience designing, developing, and supporting enterprise-scale data platforms.
At least 3 years of hands-on Databricks experience developing and operating production data engineering solutions.
Production experience with Databricks Auto Loader, including incremental cloud object storage ingestion, schema inference and evolution, schema hints, rescued data, checkpoint/state management, backfills, reprocessing, and scalable file discovery.
Production experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables), including batch and streaming pipelines, data quality expectations, quarantine or failed-record handling, streaming tables, materialized views, refresh processing, dependency management, monitoring, and alerting.
Production experience implementing Unity Catalog, including catalog, schema, and volume design; external locations; storage credentials; grants; row- and column-level access controls; lineage; and promotion across environments.
Strong hands-on Delta Lake experience, including MERGE patterns, Change Data Feed, time travel, OPTIMIZE, Z ORDER and/or liquid clustering, VACUUM and retention policies, and partitioning at scale.
Strong hands-on Apache Spark, PySpark, and Spark SQL experience, including complex transformations and production performance tuning.
Ability to diagnose and optimize Spark workloads using partition sizing, shuffle optimization, join strategy, skew handling, and Spark UI analysis.
Production Structured Streaming experience, including watermarking, late-arriving data, stateful processing, checkpointing, and exactly-once processing considerations.
Experience designing and implementing Medallion Architecture and personally building Bronze, Silver, and Gold data layers.
Experience with Databricks Workflows and deployment as code, including job dependencies, retry and failure handling, alerting, and Git-based environment promotion using Databricks Asset Bundles, Terraform, or comparable automation.
Strong understanding of distributed data processing, data optimization, scalability, and production pipeline resiliency.
Experience with production-grade data quality, error handling, monitoring, logging, observability, and pipeline recovery.
Experience with CI/CD, automated deployment, Git-based source control, branching, pull requests, and peer code reviews.
Hands-on AWS data engineering experience sufficient to analyze existing implementations and translate required functionality into a Databricks architecture.
Strong hands-on experience developing and debugging AWS Glue ETL jobs using PySpark and Python, including DynamicFrames versus DataFrames, job bookmarks, connections, job parameters, worker sizing, and runtime/version considerations.
Ability to analyze unfamiliar and undocumented AWS Glue code and determine its transformations, data movement, dependencies, and business rules.
Strong working knowledge of Amazon S3, including bucket and prefix structures, partitioning, file formats and compression, access patterns, and downstream ingestion considerations.
Experience with Amazon Redshift, including data structures, distribution and sort strategies, COPY/UNLOAD patterns, stored procedures, and identifying embedded transformation or business logic.
Strong understanding of AWS IAM and data access patterns, including roles, assumed-role relationships, policies, and authentication mechanisms.
Working knowledge of AWS Glue Crawlers and Glue Data Catalog, including schema inference, schema drift, partition management, and mapping existing catalog definitions to Unity Catalog.
Working knowledge of Redshift Spectrum and external S3-backed data access patterns.
Working knowledge of supporting AWS data, orchestration, monitoring, and security services such as Lambda, Step Functions, EventBridge, Athena, CloudWatch, and Secrets Manager.
Ability to trace end-to-end AWS data pipelines across services, identifying sources, transformations, orchestration, dependencies, access requirements, failure handling, and downstream consumers.
Experience reverse engineering undocumented legacy pipelines where business rules and processing requirements are embedded in application, ETL, orchestration, or database code.
Experience identifying pipeline behavior and dependencies when original developers or complete technical documentation are unavailable.
Experience reconciling data and validating parity between legacy and rebuilt pipelines, investigating discrepancies, and confirming equivalent business outcomes.
Ability to distinguish business logic that must be preserved from legacy technical workarounds or platform constraints.
Ability to troubleshoot complex data engineering and production issues independently while collaborating with an integrated delivery team.
Experience working in Agile delivery environments and delivering against prioritized product backlogs.
Self-motivated, detail-oriented, and organized, with the ability to work independently or in a team and in a fast-paced environment with limited supervision.
Strong communication and collaboration skills across engineering, architecture, product, and business teams.

Responsibilities

Design, develop, test, and deploy scalable Databricks data pipelines and transformation workflows for a greenfield implementation.
Reverse engineer existing AWS data processing, including undocumented code and pipeline logic, to identify transformations, business rules, dependencies, orchestration, and integration requirements.
Build, enhance, and maintain Bronze, Silver, and Gold data layers for customer data models and downstream consumption.
Design scalable ingestion using Databricks Auto Loader, including schema management, checkpointing, incremental processing, backfills, and reprocessing.
Develop and operate Lakeflow Declarative Pipelines for batch and streaming workloads, including data quality expectations, error handling, dependencies, monitoring, and recovery.
Develop batch and real-time processing solutions using Databricks, Structured Streaming, Apache Spark, PySpark, Spark SQL, and Delta Lake.
Design and optimize Delta Lake solutions using suitable MERGE patterns, Change Data Feed, partitioning or clustering, retention, and optimization strategies.
Design reliable ingestion, transformation, enrichment, and delivery of high-volume customer data.
Implement Unity Catalog governance structures, access controls, lineage, and promotion across development, test, and production environments.
Develop and support integrations between Databricks, Adobe, AWS services, and other downstream systems.
Analyze AWS Glue, Amazon S3, Amazon Redshift, Redshift Spectrum, and related services to understand existing processing, business rules, dependencies, security requirements, and target-state needs.
Establish reusable Databricks Workflows, deployment-as-code patterns, frameworks, utilities, and engineering standards.
Apply engineering best practices for data quality, performance, scalability, observability, reliability, security, and maintainability.
Tune and optimize Spark workloads, pipelines, queries, streaming processes, and data structures, including partitioning, shuffle behavior, joins, skew, and execution characteristics.
Troubleshoot data pipeline, integration, data quality, performance, and production issues; participate in root cause analysis and remediation.
Develop automated validation and testing approaches to ensure data product accuracy, completeness, and reliability.
Perform legacy-to-target parity validation, reconcile discrepancies, and confirm required business rules and data outcomes are preserved.
Distinguish required business functionality from legacy platform workarounds and avoid carrying unnecessary technical constraints into the target architecture.
Participate in design discussions, peer code reviews, technical reviews, and solution refinement.
Work within client development, CI/CD, security, governance, and deployment processes.
Create and maintain technical documentation, including data flows, implementation details, engineering patterns, operational procedures, reverse-engineering findings, and troubleshooting guidance.
Provide hands-on knowledge transfer and mentoring to client engineers to support long-term platform ownership.
Use approved AI-assisted development tools where appropriate to improve development velocity, code quality, testing, documentation, and engineering efficiency.
Identify and recommend automation and engineering improvements that increase delivery speed, solution quality, operational reliability, and production recovery.