Senior Data Engineer

Cloudious • Raleigh, NC

1d • Onsite

About The Position

We are seeking a highly skilled Senior Data Engineer to lead the migration of SQL Server databases to Databricks and Snowflake. The role involves designing and maintaining robust data infrastructure, ensuring data quality, and implementing DevOps and CI/CD best practices. The ideal candidate has extensive cloud data engineering experience, particularly with Databricks and Snowflake, and a strong understanding of healthcare data.

The position is located in Raleigh, NC, and has a stated duration of 6+ months. The job description lists 9+ years of experience with Databricks, Azure, and PySpark; its mandatory skills section also specifies 5+ years of cloud data engineering/DevOps experience.

Requirements

• At least 5 years of cloud data engineering or DevOps experience, with production experience using Databricks and Snowflake.
• Strong design experience with CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins, or GitLab CI.
• Proven experience with SQL Server-to-cloud migration.
• Proficiency in Infrastructure as Code tools such as Terraform, Bicep, or CloudFormation.
• Fluency in Python and SQL.
• Solid understanding of medallion architecture, data lineage, and governance patterns.
• Experience with Azure cloud platform or, alternatively, AWS or GCP.
• The job description also lists 9+ years of experience with Databricks, Azure, and PySpark.

Nice To Haves

• PySpark experience is strongly preferred.
• Experience with healthcare or payer data, including HEDIS, claims, and clinical data.
• Exposure to Databricks Unity Catalog and MLflow.
• Familiarity with Genie/AI-BI or similar natural-language query tools.
• Azure cloud platform experience is preferred.

Responsibilities

• Lead SQL Server-to-Databricks and Snowflake migration, including schema conversion, ETL re-platforming, data validation, and cutover planning.
• Design and maintain Databricks infrastructure, including Unity Catalog, Bronze/Silver/Gold medallion architecture, cluster and job orchestration, and cost optimization.
• Build and manage Snowflake environments, including warehouses, RBAC, data sharing, and performance tuning.
• Own CI/CD pipelines for data engineering workflows, with automated testing, deployment, and rollback.
• Implement Infrastructure as Code using Terraform or Bicep for repeatable and auditable environment provisioning.
• Establish DevOps best practices, including branching strategy, code review gates, and promotion across development, test, and production environments.
• Build monitoring, alerting, and observability for pipeline health and data quality.
• Collaborate with data engineers, analysts, and data scientists across scrum teams to support downstream needs, including HEDIS, quality, and medical economics.
• Ensure a HIPAA-compliant security posture across cloud environments, focusing on encryption, access controls, and audit logging.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed