Senior Data Engineer

Dr. Berg Nutritionals

4d•Remote

About The Position

Dr. Berg Nutritionals is a large health education and supplement company leveraging Dr. Eric Berg's popular YouTube channel. The business has significant annual revenue and is rapidly growing its subscription base. They operate a lean engineering culture with custom infrastructure primarily on Azure, using C# .NET and Microsoft Agent Framework for AI agents. Collaboration tools include Microsoft 365 and Trello. This role is central to building a unified data warehouse and AI intelligence system to provide leadership with a trustworthy view of business performance, customer value, content attribution, and operational insights. The Senior Data Engineer will be the foundational hire for this system, responsible for data pipelines from various sources into the warehouse. This is a critical role where data accuracy directly impacts business decisions, requiring a serious commitment to responsibility and technical skill.

Requirements

5–8+ years of professional data engineering experience.
At least 2–3 years working primarily in Azure (Data Factory, Synapse, Fabric, or comparable).
Strong SQL skills, including query tuning, execution plan analysis, and indexing strategy.
Production-level proficiency in C# and/or Python for custom connector work.
Demonstrated experience building pipelines against messy real-world APIs, ideally Amazon SP-API, NetSuite, or similar commerce/ERP sources.
Infrastructure-as-code experience using Bicep, Terraform, or ARM templates.
Real on-call experience, understanding the importance of good runbooks and alerting.
Strong written communication skills for documenting decisions and runbooks.

Nice To Haves

Direct experience with dbt or a comparable transformation framework.
Experience with Microsoft Fabric specifically, or a strong point of view on Fabric vs. Synapse vs. Snowflake.
Familiarity with Microsoft Agent Framework (Semantic Kernel, AutoGen) or comparable agent orchestration systems.
E-commerce or direct-to-consumer industry experience, particularly at multi-channel scale.
Experience with vector databases (Azure AI Search, pgvector, Pinecone) for AI-retrieval use cases.
Prior experience as the first or founding data engineer at a growing company.

Responsibilities

Build and maintain ingestion pipelines from source systems (Amazon SP-API, Shopify Admin API, NetSuite, Klaviyo, Recharge, YouTube Data and Analytics APIs, GA4, Google Ads, Meta Ads, Triple Whale, and ~15 others) into the data warehouse, ensuring proper error handling, idempotency, and incremental load patterns.
Own orchestration and scheduling of data pipelines, determining execution order, dependencies, and timing to meet business needs (e.g., financial data freshness, respecting API quotas).
Implement and manage monitoring, alerting, and on-call procedures for all pipelines, including health checks, data quality gates, and Azure Monitor alerts.
Optimize pipeline performance and costs by managing partitioning strategies, query tuning, incremental processing, and conducting monthly cost reviews.
Manage source system and vendor API integrations, adapting to changes in APIs and reporting structures, and handling API keys, service accounts, and rate limits.
Define and enforce data contracts between source systems and downstream consumers to ensure data integrity and prevent bad data from reaching AI systems.
Establish and maintain infrastructure-as-code practices (Bicep, Terraform) and CI/CD pipelines for data engineering changes, including environment separation (dev/staging/production).