Senior Data Engineer (AI) --100% Remote
About The Position
We are seeking a Senior Data Engineer with strong expertise in modern data engineering and working knowledge of MLOps practices to support the Knowledge Navigator platform. This individual will play a critical role in building and maintaining scalable data pipelines that feed downstream analytics and AI/ML use cases. The ideal candidate is a hands-on engineer who thrives in Azure-based environments, is experienced with large-scale data processing using Databricks, and can operate effectively within established enterprise architectures. This role requires a strong focus on data reliability, pipeline performance, and adherence to enterprise data governance and security standards, while collaborating across engineering, analytics, and AI teams.
Requirements
- 7+ years of professional experience (post-graduate) in data engineering or related field
- Strong hands-on experience with Azure data services , including: Azure Data Lake Storage (ADLS), Azure Data Factory and/or Synapse Pipelines
- Deep experience with Databricks , including Spark / PySpark development and performance optimization
- Proven experience building and maintaining enterprise-scale data pipelines
- Experience integrating data from diverse enterprise systems (databases, APIs, cloud platforms)
- Solid understanding of data architecture principles, data modeling, and ETL/ELT patterns
- Experience with pipeline orchestration, automation, and monitoring
- Working knowledge of MLOps concepts , including supporting data workflows for model training and deployment
- Strong troubleshooting and performance tuning skills in distributed environments
- Familiarity with data governance, security, and access control frameworks
- Proficiency in Python and SQL
- Strong collaboration and communication skills
Nice To Haves
- Experience supporting AI/ML or GenAI platforms and workflows
- Familiarity with CI/CD pipelines for data engineering or ML workflows
- Experience with real-time/streaming data technologies (e.g., Kafka, Azure Event Hubs)
Responsibilities
- Design, build, and maintain scalable data ingestion and processing pipelines into Azure Data Lake Storage (ADLS) and Databricks
- Implement batch and/or streaming data pipelines to support analytics and AI/ML workloads
- Develop and optimize data transformations using Spark (PySpark) and SQL within Databricks
- Integrate data from enterprise source systems (e.g., APIs, relational databases, SaaS platforms) into centralized data platforms
- Ensure high data quality, integrity, and availability for downstream consumers
- Monitor, troubleshoot, and optimize pipeline performance, reliability, and cost efficiency
- Support pipeline automation, orchestration, and scheduling using tools such as Azure Data Factory or Synapse Pipelines
- Collaborate with data scientists and ML engineers to support data and model pipeline workflows
- Implement logging, monitoring, and alerting to proactively identify issues in production pipelines
- Adhere to enterprise data governance, security, and access control policies
- Work within established architectural frameworks while contributing improvements and best practices
- Partner with cross-functional teams to ensure stable and scalable data operations
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
