Databricks Data Operations Specialist

Scicom Infrastructure Services•Atlanta, GA

About The Position

Scicom Infrastructure Services is seeking a Databricks Data Operations Specialist to support the operational stability, reliability, performance, and security of a large-scale Databricks data platform. The role is part of a multidisciplinary team that includes data engineers, architects, business analysts, governance specialists, and technical managers. The specialist monitors production data pipelines, responds to incidents, troubleshoots platform and data-processing issues, coordinates deployments, and helps ensure data products meet availability, quality, and service-level requirements. The position supports production Databricks environments and workloads using Apache Spark, Delta Lake, SQL, Python, cloud infrastructure, monitoring tools, and automated deployment practices. Responsibilities span platform and pipeline operations, incident and problem management, observability, data quality and contracts, deployment and change management, security and governance support, cost and performance management, and operational documentation and continuous improvement.

Requirements

Bachelor’s degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience.
At least four years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or a related technical role.
At least two years of hands-on experience supporting Databricks environments or production Databricks workloads.
Experience supporting production ETL, ELT, batch, or streaming data pipelines.
Working knowledge of Databricks, Apache Spark, PySpark or Python, SQL, Delta Lake, Databricks Workflows or Jobs, cloud object storage, pipeline monitoring, log analysis, and troubleshooting.
Experience investigating failed jobs, delayed pipelines, schema issues, data discrepancies, and performance problems.
Familiarity with medallion architecture and Bronze, Silver, and Gold data layers.
Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud.
Experience using ticketing and service-management tools such as ServiceNow, Jira, or a comparable platform.
Experience with monitoring, alerting, logging, or observability tools.
Understanding of incident, problem, change, and release-management processes.
Ability to write and maintain technical procedures and operational documentation.
Strong analytical, organizational, communication, and troubleshooting skills.
Ability to work effectively with engineers, business analysts, architects, security teams, and nontechnical stakeholders.
Ability to support multiple production priorities in a structured and timely manner.
Strong operational ownership, attention to detail, and ability to remain calm and organized during production incidents.
Methodical troubleshooting and root-cause analysis, including distinguishing symptoms from underlying technical causes.
Strong documentation and process-discipline skills, with ability to prioritize incidents by business and technical impact.
Collaborative approach and commitment to automation, reliability, security, continuous improvement, and ownership of issues through resolution.

Nice To Haves

Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional.
Databricks Certified Associate Developer for Apache Spark.
Microsoft Azure, AWS, or Google Cloud certification.
Experience with Unity Catalog, Delta Live Tables, Lakeflow, Auto Loader, Structured Streaming, or Databricks SQL.
Experience with Azure Data Factory, Azure Data Lake Storage, Azure Monitor, Log Analytics, AWS Glue, Amazon S3, CloudWatch, or comparable cloud services.
Experience with Terraform, Git, Azure DevOps, GitHub Actions, Jenkins, or other CI/CD tools.
Experience with Splunk, Datadog, Grafana, Prometheus, or similar observability platforms.
Experience supporting APIs, Kafka, Event Hubs, Kinesis, Airflow, dbt, Snowflake, or Power BI.
Knowledge of data contracts, schema registries, metadata management, data lineage, and data-quality frameworks.
Familiarity with OCDS, procurement data, contracting data, financial data, or public-sector data standards.
Experience supporting government, regulated, or high-security environments.
Experience operating systems with formal service-level agreements and 24-hour support requirements.
Experience with disaster recovery, backup, archival, and operational-resilience testing.

Responsibilities

Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and associated cloud services.
Provide day-to-day operational support for production Databricks environments and monitor scheduled and event-driven pipelines for failures, delays, data-quality issues, and performance degradation.
Troubleshoot Databricks jobs, Spark workloads, cluster failures, notebook errors, connectivity issues, and data-processing exceptions; restart, rerun, recover, or coordinate remediation of failed workflows.
Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads.
Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes.
Support cluster, serverless compute, instance-pool, cluster-policy, and SQL-warehouse configuration and maintenance; monitor resource utilization and recommend compute and scheduling improvements.
Maintain consistent configurations across development, testing, staging, and production environments, and support platform upgrades, runtime changes, library updates, and configuration changes.
Coordinate with Databricks support and cloud-service providers when vendor assistance is needed.
Monitor ETL, ELT, streaming, API, and file-based integration processes; validate completion within required processing windows and investigate missing, late, duplicated, incomplete, or inconsistent data.
Perform controlled data reprocessing, backfills, reconciliation, and recovery; support ingestion and transformation involving common file formats, APIs, databases, message queues, and cloud storage.
Monitor Bronze, Silver, and Gold layers; validate source-to-target counts, control totals, schema consistency, and business-rule compliance; coordinate recurring pipeline or transformation fixes with data engineers.
Maintain operational runbooks and document production interventions according to change-control procedures.
Triage production incidents based on severity, business impact, affected systems, and service-level commitments; coordinate response across engineering, cloud, security, governance, and business teams.
Provide incident status updates, escalate issues when appropriate, conduct or support root-cause analyses, document incident details and corrective actions, and participate in post-incident reviews.
Track recurring problems, recommend permanent improvements, help automate remediation, and maintain incident, problem, and service-request records in the ticketing system.
Develop and maintain dashboards, alerts, and operational reports for Databricks and associated data services; monitor job success, pipeline latency, processing duration, utilization, data freshness and quality, and compute consumption.
Configure actionable alerts, integrate monitoring data with enterprise observability platforms, define operational health indicators and service-level indicators and objectives, and report operational metrics.
Identify performance trends and risks, and maintain dashboards for availability, incident volume, recovery time, pipeline status, and data freshness.
Monitor data-contract compliance and validate schema, field, data-type, frequency, freshness, completeness, and quality requirements; detect schema drift and contract violations and coordinate resolution with relevant stakeholders.
Execute data-quality controls, track exceptions and ownership, support dataset reconciliation, assist with OCDS-structured data validation when applicable, and maintain evidence for data quality, lineage, audit, and governance.
Support releases and deployments of Databricks notebooks, jobs, workflows, libraries, configurations, and infrastructure changes across environments.
Verify deployment readiness, approvals, test results, dependencies, and rollback plans; participate in release validation and post-deployment monitoring.
Support CI/CD and infrastructure-as-code deployments; maintain deployment logs, implementation records, configuration documentation, and release notes; ensure emergency changes follow approval and documentation requirements.
Support environment comparisons, investigate configuration drift, and coordinate production changes to minimize disruption.
Support Databricks identity, access, and entitlement administration, including onboarding, offboarding, group membership, workspace permissions, service-principal access, and Unity Catalog permissions and resources.
Apply role-based and least-privilege access principles; monitor access failures, unusual activity, audit events, and policy violations; coordinate access requests with security, governance, and data owners.
Support secret management, security reviews and audits, retention, archival, backup, recovery, and disaster-recovery procedures; ensure support work complies with security and privacy requirements.
Monitor Databricks and cloud consumption; identify inefficient or misconfigured resources and analyze job duration, utilization, query performance, storage, and workload patterns.
Recommend and support compute right-sizing, autoscaling, scheduling, caching, partitioning, workload isolation, compute policies, tagging, budgets, and cost alerts; report costs and track optimization impact.
Create and maintain procedures, troubleshooting guides, escalation matrices, runbooks, configuration and dependency documentation, and an inventory of production jobs, pipelines, data products, and interfaces.
Identify opportunities to automate monitoring, support, recovery, and reporting; participate in operational-readiness reviews and ensure new production solutions have monitoring, alerts, logging, support procedures, and ownership assignments.
Share troubleshooting practices and lessons learned, and contribute to platform standards, operational policies, and continuous-improvement initiatives.