Principal Data Engineering - Hybrid
About The Position
As a Principal Data Engineering professional within the Optum Insight Engineering AI team, you will design and develop robust public cloud data systems, services, and reusable patterns that drive massive scale and performance. Our team's vision is to deliver secure, private, and highly scalable Azure cloud solutions that enable the organization to utilize clean, reliable data safely and swiftly. In this role, you will lead the creation of modern ETL/ELT pipelines, Spark workflows, and AI integrations. Working with advanced platforms such as Azure Databricks, Snowflake, and LLMs, you will build data solutions that empower machine learning, insights generation, and business automation, directly contributing to better clinical and administrative connectivity across the healthcare system. If you are located in Eden Prairie, you will have the flexibility to work remotely, as well as work in the office as you take on some tough challenges. This position follows a hybrid schedule with four in-office days per week.
Requirements
- Bachelor’s degree or equivalent experience (such as an additional 8+ years of data engineering experience)
- 10+ years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes
- 7+ years of experience in Python
- 5+ years of experience in Apache Spark (PySpark/Spark SQL)
- 5+ years of experience in SQL, including designing complex data schemas and query performance optimization
- 3+ years of experience with API design and lifecycle management (GraphQL, REST, etc.)
- 3+ years of experience building and deploying cloud-based solutions using Azure Databricks with UC, Snowflake, Functions, or Service Bus
- 3+ years of experience with DevOps automation using Terraform
- 3+ years of experience with CI/CD processes and tools (such as GitHub Actions, GIT, Artifactory, or Sonar)
- 2+ years of experience building LLM integrations for workflow automation or business needs
Nice To Haves
- Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related discipline
- Healthcare and Provider domain experience
- Experience working with LLMs
- Extensive knowledge of data architecture principles (e.g., Data Lake, Databricks Delta Lake, Data Warehousing, etc.)
- Extensive knowledge of data modeling techniques including slowly changing dimensions, aggregation, partitioning, and indexing strategies
- Proven ability to independently troubleshoot and performance tune large-scale enterprise systems
- Proven excellent collaborator with experience working effectively with cross-functional teams such as leadership, product management, and engineering, with a willingness to inspire other data engineers, data scientists, and analysts
- Proven solid communication skills with the ability to communicate technical concepts to both technical and non-technical audiences
Responsibilities
- Design and Develop Scalable Cloud Applications: Develop services, controls, and reusable patterns (such as microservices and Azure functions) that enable the team to deliver value safely, quickly, and sustainably in the Azure public cloud while enabling security and privacy at scale
- Build and Optimize Large-Scale Data Pipelines: Design, build, optimize, and manage modern large-scale data pipelines and ETL/ELT processing on Azure Databricks, LakeBase, and Apache Spark to support data integration, analytics, machine learning features, and predictive modeling
- Deploy AI and Data-Driven Solutions: Develop and deploy large-scale data pipelines empowering machine learning algorithms, insights generation, business intelligence dashboards, reporting, and new data products while utilizing enterprise-approved AI tools to address complex business challenges
- Develop AI-Powered Business Solutions: Build AI-based solutions for solving business needs, automating processes, and streamlining workflows to drive operational efficiency and continuous improvement
- Architectural Evolution and Standards: Participate in the architectural evolution of data engineering patterns, frameworks, systems, and platforms, including defining best practices and standards for managing data collections and integrations
- Improve System Quality and Data Reliability: Write advanced, complex SQL with performance tuning and optimization to identify and implement ways to improve data reliability, data integrity, system efficiency, and overall quality
- Collaborative Leadership and Mentoring: Foster high-performance, collaborative technical work resulting in high-quality output. Mentor other data engineers, providing technical direction and training on leveraging cloud data platforms
- Stakeholder and Requirement Analysis: Intersect skillfully with business stakeholders and third-party technical organizations to understand new product capabilities, decompose implementations into specific functional changes, analyze data for decision-making, and provide detailed, realistic estimates
- Evaluate Emerging Trends: Evaluate emerging trends to inform solution design, strategic innovation, and the evolution of cloud data architectures
- Best Practices in performance scalability and optimization
Benefits
- comprehensive benefits package
- incentive and recognition programs
- equity stock purchase
- 401k contribution
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
