Microsoft Fabric Data Engineer

The AES Group•Indianapolis, IN

4d•Onsite

About The Position

The Data Engineer is a hands-on builder responsible for developing data pipelines, API integrations, and AI infrastructure that bring structured and unstructured data into a governed Azure-based architecture. This delivery-focused role requires designing, coding, testing, and maintaining production-ready solutions across both AI document ingestion and structured ETL/ELT data engineering tracks.

Requirements

5+ years of hands-on data engineering or backend engineering experience delivering production-grade data pipelines, integrations, and relational data models.
Strong experience building API integrations and data connectors using REST APIs, OAuth, webhooks, and modern integration frameworks; experience with MCP or similar protocols preferred.
Experience designing and implementing document ingestion and AI search solutions, including OCR, chunking, embeddings, vector databases, and retrieval-augmented generation (RAG) using Azure OpenAI, OpenAI, Claude, or similar technologies.
Proven expertise delivering production workloads on Microsoft Azure, including Azure Fabric (Lakehouse, Data Factory, Pipelines, Delta Lake), Azure Functions, Storage, and AI Services.
Hands-on AWS experience supporting data engineering solutions using services such as S3, Lambda, API Gateway, Glue, and RDS/Aurora.
Strong proficiency in Python and/or PySpark for pipeline orchestration, data transformation, API development, testing, and CI/CD automation.
Experience designing and supporting ETL/ELT pipelines with data validation, quality controls, monitoring, error handling, and operational reliability.
Advanced knowledge of PostgreSQL, relational database design, schema modeling, query optimization, and medallion architecture (Bronze/Silver/Gold) data platforms.
Experience implementing data lineage, auditability, and governance controls within regulated or compliance-sensitive environments, including familiarity with ALCOA+ principles.

Nice To Haves

Experience in pharmaceutical, biotechnology, or medical device development environments —working knowledge of regulated data requirements is a strong differentiator
Familiarity with GxP data integrity requirements (21 CFR Part 11, ALCOA+) and their practical implications for data pipeline design, electronic records, and audit trail implementation
Direct experience with any DDCS or WBWD systems: Oracle Agile PLM / Siemens Teamcenter, LabVantage LIMS, Darwin, Veeva Vault/QDocs, SmartLab/Biovia LES, NuGenesis NG9
Microsoft Azure Data Engineer Associate or AWS Certified Data Engineer certification
Experience with infrastructure-as-code (Bicep, Terraform) and GitOps deployment patterns for Azure-native data workloads
Familiarity with LangChain, semantic chunking strategies, or embedding model selection for domain-specific scientific document retrieval
Experience with Microsoft Fabric Unity Catalog, Delta Sharing, or governed data sharing patterns across workspaces
Experience with Docker and Kubernetes-based orchestration for data pipeline services in production environments

Responsibilities

Design, build, and maintain data pipelines that integrate laboratory, quality, design, and partner data into Azure Fabric Lakehouse and PostgreSQL using a Bronze-Silver-Gold architecture.
Develop ETL/ELT processes with data validation, harmonization, auditability, data quality controls, and regulatory-compliant lineage from source to governed datasets.
Build AI-ready document ingestion pipelines and MCP connectors for systems such as SharePoint, Qdocs/Veeva, Jama, and TurboAC, supporting semantic search, retrieval, and enterprise LLM applications.
Develop partner data ingestion workflows leveraging OCR, LLM-based extraction, confidence scoring, and exception handling for external CMO/CRO documentation.
Implement monitoring, alerting, operational documentation, and data lineage across all pipelines and integrations.
Develop and maintain RESTful API and instrument data connectors with secure authentication, error handling, schema management, and production-grade reliability.
Integrate Azure OpenAI (or equivalent) services into retrieval-augmented generation (RAG) solutions using a vendor-agnostic architecture.
Build and support cloud-native data solutions across Azure Fabric and AWS, including containerized services, CI/CD pipelines, infrastructure security, secrets management, and cross-cloud data movement.
Implement and maintain PostgreSQL and medallion-layer data models, including harmonization logic, referential integrity, lineage tracking, and governed data standards.
Partner with Data Architects and WBWD stakeholders to evolve schemas, data structures, and integration patterns as business requirements mature.