Onshore - Sr Engg_CurationSup

Fractal Analytics•California, CA

1d•$150,000 - $170,000•Remote

About The Position

We are looking for a Digital Operations Engineer with strong expertise in data engineering, Azure cloud, and data quality frameworks. The role focuses on building scalable data platforms, ensuring high data reliability and quality, and enabling seamless operations through automation and monitoring. The ideal candidate will have hands-on experience with Databricks, Python, SQL, and a strong foundation in data quality and governance practices.

Requirements

Strong experience in Python and SQL
Hands-on experience with Azure Databricks (Spark, Delta Lake)
Experience with Azure data services (ADF, ADLS, Synapse)
Solid understanding of data engineering concepts (ETL/ELT, data modeling, warehousing)
Proven experience in data quality frameworks, tools, and methodologies
Experience with DevOps tools (Azure DevOps, Git, CI/CD pipelines)
Experience implementing data validation, profiling, and cleansing techniques
Knowledge of tools/frameworks like Great Expectations, Deequ, or custom validation frameworks
Understanding of data governance, lineage, and metadata management
Strong troubleshooting and debugging capabilities
Experience in production support and incident handling
Familiarity with monitoring and logging tools

Responsibilities

Design, develop, and maintain robust data pipelines using Azure Databricks, Python, and SQL
Build and optimize ETL/ELT processes for batch and real-time data
Implement scalable solutions using Spark and distributed data processing frameworks
Develop reusable components and standardized data engineering practices
Design and implement data quality frameworks (validation rules, anomaly detection, profiling)
Establish data quality KPIs such as completeness, accuracy, consistency, and timeliness
Build data validation pipelines and automated checks within ETL processes
Perform root cause analysis for data issues and implement corrective/preventive actions
Collaborate with business teams to define data standards and governance policies
Work on Azure ecosystem: Azure Data Factory, ADLS Gen2, Synapse Analytics, Azure Databricks
Optimize Delta Lake architectures and ensure efficient data storage and access
Implement secure and scalable data lake/lakehouse solutions
Monitor data pipelines and workflows for failures, latency, and performance issues
Ensure production stability, SLA adherence, and incident management
Set up monitoring, logging, and alerting (Azure Monitor, Log Analytics)
Support data reconciliation and audit processes
Implement CI/CD pipelines using Azure DevOps and Git
Automate deployment of data pipelines and infrastructure (ARM templates/Terraform)
Integrate automated data quality checks into CI/CD workflows
Ensure proper versioning and release management practices