AI & Data Engineer, Data Discovery Services
About The Position
Join the Data Discovery Services team within Enterprise Data Platforms, where we deliver and maintain the data foundation platforms that power discovery, search, and data accessibility across the Bristol Myers Squibb enterprise. We run multiple search and discovery services used across the company -- and we're building the next generation of AI-powered discovery on top of them. Our work makes enterprise data findable, accessible, and actionable for teams across the organization. At the core of this is a semantic knowledge layer -- metadata, taxonomies, and relationships that describe what data means and how it connects -- curated as a data inventory that helps AI work reliably across the enterprise. This is a high-impact team where engineering, search, and applied AI come together to solve real problems at scale. As an AI / Data Engineer, you'll be a hands-on Python developer building the pipelines and integrations that make enterprise data more discoverable. Your primary focus is data engineering -- pipelines, metadata enrichment, transformations, and platform integrations. You'll also contribute to search and AI-powered retrieval as you grow into the role. Working alongside data engineers, search engineers, and data scientists, this is a hands-on engineering role -- you'll write code, build pipelines, ship features, and own what you deliver.
Requirements
- Bachelor’s degree in computer science, Data Science, Information Science, Engineering, or a related field. Master's degree preferred.
- Demonstrated proficiency in data engineering, software engineering, or a related technical discipline, with a track record of delivering production data pipelines.
- Proficient Python skills -- this is your primary language day-to-day.
- Proficiency in Structured Query Language (SQL).
- Experience with Databricks and AWS Glue for data pipelines and transformations.
- Solid data engineering fundamentals: extract-transform-load (ETL/ELT) patterns, data modeling, data quality, and pipeline orchestration.
- Familiarity with AWS cloud services (S3, Lambda, API Gateway, Glue).
- Experience with OpenSearch or Elasticsearch.
- Understanding of metadata management and data cataloging concepts.
- Effective problem-solving skills and willingness to learn.
- Good communication skills and ability to work collaboratively in a team.
Nice To Haves
- Experience with semantic knowledge layers, RAG patterns, vector search technologies, and building AI-ready data inventories.
- Familiarity with semantic search, embeddings, chunking strategies, relevance tuning, and semantic knowledge metadata (entity relationships, taxonomies, context enrichment).
- Exposure to AI agent patterns, MCP, or large language model orchestration frameworks.
- Experience with ontology or taxonomy technologies (such as Turtle, Resource Description Framework, Web Ontology Language, or SPARQL query language) or management platforms.
- Familiarity with graph databases or knowledge graph technologies.
- Experience with metadata enrichment, data lineage, or data quality frameworks.
- Exposure to Azure OpenAI, Google Vertex AI, or Amazon Bedrock.
- Experience with Docker, Elastic Container Service (ECS), or CloudFormation.
- Prior exposure to pharma or life sciences.
Responsibilities
- Build and maintain Python pipelines that pull metadata from enterprise data catalogs, enrich it with taxonomy tags and ownership information, and publish it to the discovery platform.
- Tune and optimize search indexes -- adjust analyzers, boost fields, and test queries -- to ensure results match what users need.
- Build a semantic knowledge layer -- chunking documents, generating vector embeddings, and enriching them with semantic knowledge metadata -- to grow a data inventory that supports retrieval-augmented generation (RAG) and helps AI systems and large language models (LLMs) find and use the right context.
- Maintain integrations that sync ontology and taxonomy changes into the discovery platform, so classifications stay current.
- Investigate and resolve data pipeline issues across Databricks and AWS Glue, trace root causes through metadata enrichment flows, and add data quality checks to prevent recurrence.
- Build API endpoints and Model Context Protocol (MCP) servers that expose search and metadata capabilities to applications and AI agents.
- Design metadata pipelines that map cross-domain dataset relationships and add them to the cross-domain join catalog with confidence scores.
- Analyze search patterns, capture user feedback, and improve the discovery experience so the system learns and improves over time.
Benefits
- wellbeing support
- retirement and financial protection benefits
- insurance offerings (medical, dental, vision, life and disability)
- Flexible Time Off (FTO)
- 11 paid company holidays each year
- 160 hours of paid vacation annually
- 3 optional holidays
- paid sick leave
- up to two paid volunteer days per year
- summer hours flexibility
- leaves of absence for medical, personal, parental, caregiver, bereavement, or military needs
- annual Global Shutdown between Christmas Day and New Year's Day
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
