Lead Big Data Engineer

AT&T • Dallas, TX

379d • Onsite

About The Position

The Lead Big Data Engineer at AT&T Services, Inc. is responsible for providing Application Security Log Alarming (ASLA) and Advanced Secure Telecommunications Routing Architecture (ASTRA) support in cloud environments. This role involves designing, deploying, and managing various Platform as a Service (PaaS) and Software as a Service (SaaS) platforms, particularly Databricks and Snowflake, while ensuring data integrity and security. The engineer will work on data analysis, validation, and the implementation of data quality procedures, supporting Data Scientists in data sourcing and preparation. Additionally, the position requires collaboration with policy and security teams to develop data policies and retention models, as well as optimizing cloud costs and managing cloud-based environments.

Requirements

• Bachelor's degree in Electronic Engineering, Engineering, or Computer Science, or foreign equivalent.
• 2 years of experience in the job offered or a related occupation.
• Experience with Azure cloud design, development, testing, and administration.
• Experience defining network solutions for cloud-based environments.
• Experience with Azure Keyvault, secret scopes, private endpoints, Azure AD, and Active Directory services.
• Experience with HDI, Azure's cloud-based implementation of Hadoop, ADLSg2, Delta Live Tables, and pipelines.
• Experience managing and configuring Databricks on Azure.
• Experience designing and configuring real-time UI monitoring with Prometheus and Grafana.

Responsibilities

• Provide Application Security Log Alarming (ASLA) and Advanced Secure Telecommunications Routing Architecture (ASTRA) support in the cloud.
• Design, deploy, and manage PaaS and SaaS platforms, including Databricks and Snowflake.
• Define data requirements and gather large volumes of structured and unstructured data.
• Validate data using various data tools in the Big Data Environment.
• Support standardization, customization, and ad-hoc data analysis.
• Develop mechanisms to ingest, analyze, validate, normalize, and clean data.
• Implement statistical data quality procedures on new data sources.
• Support Data Scientists in data sourcing and preparation for visualization and insights.
• Collaborate with Big Data Policy, security, and legal teams to create data policies and retention models.
• Develop and maintain data engineering best practices and contribute to data analytics insights.
• Drive down cloud costs for job runs using GPUs instead of CPUs.
• Determine potential deployment and use of ICEBerg storage across platforms.
• Utilize Azure cloud experience for designing and managing cloud-based environments.
• Define network solutions and structures for cloud-based solutions.
• Manage and configure Databricks on Azure to ensure platform integrity.
• Design and configure real-time UI monitoring using Prometheus and Grafana.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Industry: Telecommunications
• Education Level: Bachelor's degree