Lead Data/AI Engineering
About The Position
This role involves designing, developing, and maintaining scalable and efficient data pipelines using advanced technologies like Python, PySpark, and Databricks. The position requires integrating data from diverse sources while ensuring high standards of data quality, consistency, and reliability. A key responsibility is formulating and implementing comprehensive data architecture strategies, including data modeling, schema design, and data storage solutions, as well as optimizing data processing workflows for enhanced performance, scalability, and cost-efficiency. Collaboration with data scientists, analysts, and stakeholders is essential to understand data requirements and deliver tailored data solutions. The role also includes identifying and resolving data-related issues, supporting data infrastructure, and maintaining detailed documentation of all data pipelines, architecture, and processes. Additionally, the position drives the design and implementation of data models to enhance business decision-making by generating insights from internal and external data assets, defining data requirements, and mining and validating large-scale structured and unstructured datasets using cloud-based tools. The role supports both standard and customized data analyses, develops robust mechanisms for data ingestion, analysis, validation, normalization, and cleaning, and upholds best practices in data engineering while contributing to advanced data analytics and visualization initiatives.
Requirements
- Requires a Bachelor’s degree, or foreign equivalent degree in Electrical and Electronic Engineering, Computer Science, or Information Technology
- Five (5) years of progressive, post-baccalaureate experience in the job offered or five (5) years of progressive, post-baccalaureate experience in a related occupation
- Utilizing programming languages (Python, PySpark, and Scala)
- Using data warehousing solutions including Snowflake
- Utilizing advanced SQL skills
- Working with Databricks within Azure cloud environments
- Employing big data technologies, including Apache Spark, Hadoop, and Kafka
- Employing data science concepts and machine learning
Responsibilities
- Design, develop, and maintain scalable and efficient data pipelines utilizing advanced technologies such as Python, PySpark, and Databricks.
- Integrate data from diverse sources while ensuring high standards of data quality, consistency, and reliability.
- Formulate and implement comprehensive data architecture strategies, encompass data modeling, schema design, and data storage solutions.
- Optimize data processing workflows for enhanced performance, scalability, and cost-efficiency.
- Collaborate with data scientists, analysts, and stakeholders to accurately understand data requirements and deliver tailored data solutions.
- Identify and resolve data-related issues, support data infrastructure, and maintain detailed documentation of all data pipelines, architecture, and processes.
- Drive the design and implementation of data models to enhance business decision-making by generating insights from both internal and external data assets.
- Define data requirements, mine and validate large-scale structured and unstructured datasets using cloud-based tools and supporting both standard and customized data analyses.
- Develop robust mechanisms for data ingestion, analysis, validation, normalization, and cleaning.
- Uphold best practices in data engineering and contributing to advanced data analytics and visualization initiatives.
Benefits
- Medical/Dental/Vision coverage
- 401(k) plan
- Tuition reimbursement program
- Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
- Paid Parental Leave
- Paid Caregiver Leave
- Additional sick leave beyond what state and local law require may be available but is unprotected
- Adoption Reimbursement
- Disability Benefits (short term and long term)
- Life and Accidental Death Insurance
- Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
- Employee Assistance Programs (EAP)
- Extensive employee wellness programs
- Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
