Senior Azure Data Engineer

USEReady Technology

About The Position

USEReady is seeking an experienced Senior Azure Data Engineer to design, develop, optimize, and maintain scalable data engineering solutions on Azure. The role focuses on building high-performance data pipelines and reliable data platforms that support analytics and business intelligence initiatives, using technologies including Azure Databricks, Azure Data Factory (ADF), PySpark, Python, and SQL. The engineer will work with Azure data services and modern lakehouse technologies to develop data processing solutions, maintain Delta Lake tables and data models, implement governance and quality controls, and improve workload performance, scalability, and cost efficiency. The role also involves troubleshooting operational issues, collaborating with stakeholders and technical teams, automating workflows, and mentoring junior data engineers. USEReady helps enterprises modernize data and analytics environments and build AI-ready systems. Founded in 2011 and headquartered in New York City, the company has more than 450 experts across the United States, Canada, India, and Singapore, and serves industries including financial services, healthcare, manufacturing, government, education, and retail.

Requirements

At least 6 years of experience in data engineering, with strong experience in Azure-based data platforms.
Strong hands-on experience with Azure Databricks.
Expertise in Databricks Notebooks using Python and SQL.
Strong experience with Databricks Unity Catalog, governance, security, and access control.
Hands-on experience with Databricks performance optimization and cost/compute optimization.
Strong experience with Azure Data Factory and ETL/ELT orchestration.
Advanced Python and PySpark programming skills.
Strong understanding of Apache Spark architecture, internals, and performance tuning.
Strong SQL skills, including complex queries, query optimization, joins, aggregations, and large-volume data processing.
Strong understanding of data warehousing concepts and data modeling, including Star and Snowflake schemas.
Experience with Delta Lake and modern lakehouse architecture.
Good knowledge of Azure Data Lake Storage (ADLS).
Experience with Azure Synapse Analytics and Azure SQL Database.
Understanding of data governance, data security, access management, and data lineage.
Experience working with large-scale distributed data processing environments.
Strong problem-solving, communication, and collaboration skills.
A Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field is preferred.

Nice To Haves

Databricks certifications.
Microsoft Azure Data Engineering certifications.
Experience working with enterprise-scale Azure data platforms.
Experience implementing data quality, governance, and security frameworks.
Experience mentoring or technically guiding junior data engineers.

Responsibilities

Design, develop, and maintain scalable, reliable data pipelines using Azure Databricks and Azure Data Factory (ADF).
Develop data processing solutions using Python, PySpark, SQL, and Apache Spark.
Build and maintain Delta Lake tables and scalable data models for analytics and business intelligence workloads.
Implement and manage Databricks Unity Catalog for data governance, security, access control, and data discovery.
Optimize Databricks jobs, Spark workloads, SQL queries, and data pipelines for performance, scalability, and cost efficiency.
Apply Apache Spark internals and tuning techniques to improve distributed data processing performance.
Develop complex, optimized SQL queries, including efficient joins and processing of large datasets.
Design and implement ETL/ELT workflows using Azure Data Factory.
Work with Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL Database.
Implement data quality checks, validation, reconciliation, monitoring, and error-handling mechanisms.
Implement data governance, security, access control, and data lineage across data platforms.
Troubleshoot pipeline failures and performance issues and ensure data solutions are highly available and reliable.
Collaborate with business stakeholders, data architects, analysts, and engineering teams to understand requirements and deliver data solutions.
Automate manual processes and continually improve data engineering workflows and operational efficiency.
Mentor junior data engineers and promote Azure and Databricks data engineering best practices.