Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)

Southern Company • Birmingham, AL

5d

About The Position

Southern Company’s Operations Data Analytics team supports the data foundations and integration capabilities needed for reliable analytics and reporting across multiple organizations. This Data Engineer role focuses on Safety data, creating data pipelines, curating source schemas, and supporting data product development in the Azure Databricks lakehouse platform. The role helps create secure, reusable, high-performing data services aligned with business needs, Technology Organization strategy, and enterprise standards. It supports multiple business units and lakehouse environments, with data provisioned through Unity Catalog, to establish a unified foundation for safety and human performance data.

The engineer designs, develops, implements, integrates, and supports cloud and data solutions, with hands-on responsibility for pipelines, data models, ETL processes, lakehouse capabilities, and data storage layers. The role partners with business and technology stakeholders, developers, and infrastructure teams to align solutions across platforms; serves as a subject matter expert for data architecture and Databricks; supports operational issue resolution; and ensures project documentation and change management requirements are met.

Requirements

• Bachelor’s degree required, preferably in Computer Science, Management Information Systems, Engineering, or another technology-related field.
• Experience with Databricks, including notebooks, clusters, and jobs.
• Experience with a major cloud platform, preferably Microsoft Azure, and services including Cosmos DB, Storage Application Services, and Azure Data Factory.
• Experience architecting data-oriented solutions using cloud data architecture platforms and patterns, preferably Azure.
• Experience implementing lakehouses with medallion architecture, data engineering pipelines, and data storage layers.
• Experience with dynamic metadata-driven frameworks and large-scale data ingestion patterns.
• Experience developing complex logical and physical data models.
• Experience optimizing cloud PaaS resources to follow best practices and reduce cost, as well as tuning lakehouse performance using partitioning, aggregations, materialized views, or similar techniques.
• Experience with Agile, DevOps, and CI/CD methodologies.
• Familiarity with Apache Spark, Delta Lake, and MLflow, and knowledge of structured and unstructured data across databases, data warehouses, and data lakes.
• Knowledge of software development technologies, methodologies, and associated tools.
• Excellent root cause analysis and problem-solving skills.

Nice To Haves

• Hands-on experience with Microsoft Azure data services and cloud PaaS architecture and resources.
• Familiarity with streaming datasets and Databricks Delta Live Tables.
• Strong technical writing skills.

Responsibilities

• Design and develop robust data pipelines to ingest, store, and process data from multiple sources for analytics and reporting.
• Create and maintain logical and physical data models for efficient querying, analysis, and visualization.
• Build and maintain ETL processes that move data between systems while supporting data quality and consistency.
• Implement and support lakehouse solutions using medallion architecture, including data engineering pipelines and data storage layers.
• Optimize data lake storage, retrieval, performance, and resource utilization for analytical workloads.
• Develop dynamic, metadata-driven frameworks and scalable data ingestion patterns.
• Support streaming datasets and Databricks Delta Live Tables.
• Apply lakehouse performance-tuning techniques such as partitioning, aggregations, and materialized views.
• Architect, develop, implement, integrate, and support cloud technology solutions that meet business partner expectations for quality, cost, and timeliness.
• Work with developers and infrastructure teams to optimize application code, create reusable services, plan capacity, and protect application and data security.
• Serve as the team’s subject matter expert for data architecture and Databricks.
• Communicate effectively and promptly with customers, team members, and management, and build strong relationships with peers and business partners.
• Develop project documentation, follow Change Management policies and procedures, and respond quickly and effectively to operational problems.
• Build an understanding of the company’s business challenges and goals and exhibit Our Values.

Benefits

• Competitive base salary.
• Annual incentive awards for eligible employees.
• Health, welfare, and retirement benefits designed to support physical, financial, and emotional/social well-being.
• Potential eligibility for additional compensation, such as an incentive program, subject to applicable plan terms and conditions.

What This Job Offers

• Job Type: Full-time