Principal System Engineering - DevOps - Spark and Azure
AT&T - Dallas, TX
posted about 1 month ago
About the position
Join AT&T and reimagine the communications and technologies that connect the world. The Chief Information Office is responsible for advancing information technology performance and delivering solutions with a focus on maximizing ROI, increasing efficiency and enhancing the experience of end users. Guided by experienced leaders, Corporate Systems seamlessly integrate with advanced Technology and Operations to drive our enterprise forward. Our Systems Reliability and Software Delivery teams are unwavering in their commitment to excellence, ensuring every solution is robust and efficient. When you step into a career with AT&T, you won't just imagine the future-you'll create it. We are seeking a highly skilled and experienced professional to join our team as an Apache Spark Expert within an Azure Synapse Environment. The ideal candidate will possess deep expertise in Apache Spark and be proficient in working with Databricks. The candidate must have a strong command of Spark job definitions, PySpark, and structured streaming. In addition, the role requires comprehensive knowledge of partitioning, SQL, and Delta Lake capabilities. The successful candidate will also have hands-on experience with EventHubs, Kafka, and ADLSv2 storage, as well as MongoDB. Familiarity with graph databases, preferably Gremlin, is essential. The candidate will be responsible for leveraging these technologies to drive data processing and analytical solutions within our organization. Additionally, the ideal candidate should possess strong data science skills, including expertise in time series analysis, graph algorithms, classification, and clustering/NLP. They should also have a deep knowledge of distributed systems, particularly related to Spark, including partitioning for performance and reliability, checkpointing, architecture and provisioning of drivers and executors, sizing of Spark pools, and tuning Spark configurations. Experience in Azure architecture and developing DevOps pipelines is also required. The candidate should be proficient with Git for version control and collaboration. Moreover, they should have a solid foundation in general computer science skills, including reliability engineering, and be proficient with Kubernetes, Docker, and Helm, including developing and deploying applications on Kubernetes. A strong knowledge of Linux and Linux CLI, as well as programming skills in Java and/or Go, are also necessary. Furthermore, the candidate should have Open Telemetry developer knowledge, including experience with protobufs, receivers, collectors, and sampling. Familiarity with MLflow internals is also highly desirable. The successful candidate will bring a combination of advanced technical skills, a strong background in data science and distributed systems, and a solid understanding of DevOps and general computer science principles. This role offers an opportunity to work on cutting-edge technologies and contribute to the development of scalable data solutions within our organization.
Requirements
- 8+ years experience
- Expert level experience in Apache Spark, PySpark, SQL, and Delta Lake capabilities, EventHubs, Kafka, and ADLSv2 storage, as well as MongoDB
- Familiarity with graph databases, preferably Gremlin
- Azure Synapse Environment, Azure architecture and developing DevOps pipelines
- Strong data science skills, including expertise in time series analysis, graph algorithms, classification, and clustering/NLP
- Kubernetes, Docker, and Helm, including developing and deploying applications on Kubernetes
Nice-to-haves
- Bachelor's Degree in Computer Science, Engineering or Mathematics
Benefits
- Medical/Dental/Vision coverage
- 401(k) plan
- Tuition reimbursement program
- Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
- Paid Parental Leave
- Paid Caregiver Leave
- Additional sick leave beyond what state and local law require may be available but is unprotected
- Adoption Reimbursement
- Disability Benefits (short term and long term)
- Life and Accidental Death Insurance
- Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
- Employee Assistance Programs (EAP)
- Extensive employee wellness programs
- Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone
