Sr. Azure Data Engineer

TalentOla•Chicago, IL

1d•Hybrid

About The Position

The Sr. Azure Data Engineer develops and manages cloud data solutions using Lakehouse Architecture, Delta Lake, and Medallion (Bronze, Silver, Gold) frameworks. The role creates and maintains dimensional and normalized data models, builds large-scale data pipelines and batch-processing solutions, and supports analytics and reporting needs. It also establishes data quality checks, validation, monitoring, and reconciliation processes, while optimizing performance, scalability, reliability, and cost efficiency. The position collaborates with Data Architects, Business Analysts, BI Developers, and other stakeholders to translate business requirements into technical solutions. It participates in reviews, testing, deployment, and production support, develops reusable engineering frameworks and best practices, troubleshoots data and performance issues, and ensures compliance with governance, security, and regulatory requirements. A core focus is using agentic IDEs and AI coding agents effectively in complex, multi-module, cross-service, and legacy-modernization work. The role defines architectural intent and executable tasks for agents; establishes autonomy guardrails, permissions, review practices, and quality checks; integrates agentic workflows into CI/CD; mentors team members; and evaluates effects on the SDLC, security, compliance, auditability, traceability, and technical debt. The role is based in Chicago, Illinois, with three days per week in the office, and is described as a C2H role. Candidates must be authorized to work in the United States without current or future visa sponsorship.

Requirements

At least 10 years of data engineering experience, with a strong focus on modern cloud data platforms.
Hands-on experience with Databricks, PySpark, SQL, and Python.
Strong understanding of ETL/ELT design patterns, data integration, and data warehousing concepts.
Experience with relational databases, data modeling, partitioning, indexing, and query optimization.
Expertise building and maintaining large-scale data pipelines and batch-processing solutions.
Experience with cloud platforms such as Azure, AWS, or GCP.
Strong problem-solving, debugging, and analytical skills.
Proven experience architecting and delivering systems using agentic IDEs.
Ability to define architectural intent for agents and break features into agent-executable tasks.
Experience governing AI autonomy through guardrails, permissions, and reviews, and integrating agentic workflows into CI/CD pipelines.
Experience supervising AI agents across multi-service systems, legacy modernization, and large codebases or monorepos.
Understanding of security implications of autonomous code execution, compliance, auditability, traceability, and AI-assisted SDLC operating models.
Engineering degree: BE, ME, BTech, MTech, BSc, or MSc.
Authorized to work in the United States without requiring current or future visa sponsorship.

Nice To Haves

Technical certifications in multiple technologies are desirable.

Responsibilities

Develop and manage data solutions using Lakehouse Architecture, Delta Lake, and Medallion (Bronze, Silver, Gold) frameworks.
Create and maintain dimensional and normalized data models for analytics and reporting.
Implement data quality checks, validation frameworks, monitoring, and reconciliation processes.
Optimize data processing jobs for performance, scalability, reliability, and cost efficiency.
Collaborate with Data Architects, Business Analysts, BI Developers, and stakeholders to translate business requirements into technical solutions.
Participate in code reviews, testing, deployment, and production support.
Develop reusable frameworks and best practices for data engineering and platform development.
Troubleshoot and resolve performance bottlenecks, data issues, and production incidents.
Ensure compliance with data governance, security, and regulatory requirements.
Guide effective use of agentic IDEs for complex, multi-module, or cross-service changes.
Establish review practices and quality checks for AI-generated code.
Mentor team members on balancing autonomy, correctness, and maintainability in AI-assisted development.
Design system architectures that support AI-augmented and agentic development workflows.
Define guardrails, standards, and governance for autonomous coding agents.
Evaluate the impact of agentic IDEs on the SDLC, CI/CD pipelines, security posture, and technical debt.
Define architectural intent agents can follow and break features into agent-executable tasks.
Govern AI autonomy through guardrails, permissions, and reviews, and integrate agentic workflows into CI/CD pipelines.
Supervise AI agents across multi-service systems, legacy modernization, and large codebases or monorepos.