Senior Data Engineer (Customer Data Products)

NBME•Philadelphia, PA

1d•$127 - $160•Hybrid

About The Position

NBME is seeking a Senior Data Engineer to join its growing team of data engineers and BI developers. The role supports modernization, expansion, and optimization of NBME’s AWS data platform by building data lakes, complex data integration pipelines, and scalable data solutions for analytics, AI/ML, and business decision-making. The engineer will apply AI-assisted engineering practices to improve development efficiency and share best practices across the IT organization. This hybrid role may require occasional onsite work for collaboration or meetings. NBME is prioritizing applicants within 50 miles of its Philadelphia office; onsite interviews and onboarding may be required, with advance notice where applicable. The position involves developing and operating cloud-based data engineering processes, feature stores, vector databases, and automated data pipelines. The engineer will work with data scientists, AI researchers, architects, and business stakeholders to prepare data for AI/ML, explore emerging data and MLOps technologies, promote modern engineering practices, and help ensure secure, accurate, high-quality corporate data.

Requirements

Bachelor’s degree.
At least 7 years of application development experience; internship experience does not apply.
At least 4 years of experience with big data technologies.
At least 4 years of cloud computing experience using AWS.
At least 4 years of application development experience using Python, SQL, Scala, or Java.
At least 4 years of experience with distributed data/computing tools such as MapReduce, Hadoop, Hive, EMR, Kafka, Spark, or MySQL.
At least 4 years of experience with real-time data and streaming applications.
At least 4 years of experience with NoSQL implementation, such as Mongo or Cassandra.
At least 4 years of data warehousing experience with Redshift.
At least 6 years of experience with UNIX/Linux, including basic commands and shell scripting.
At least 7 years of experience with Agile engineering practices.
At least 7 years of experience with SQL optimization.
At least 4 years of experience with PySpark.
At least 3 years of experience with process orchestration tools such as AirFlow, KubeFlow, AWS Step Functions, or Luigi.

Nice To Haves

Proven experience implementing Generative AI, LLM data preparation pipelines, and vector databases such as Pinecone, Milvus, or pgvector.
Strong experience building and maintaining feature stores for machine learning models.
Experience building highly scalable, secure, production-ready APIs and Data-as-a-Service (DaaS) platforms.
AWS Certified Data Engineer or AWS Certified Machine Learning – Specialty certification.
At least 3 years of machine learning experience.
Experience building a Data-as-a-Service platform.
Experience building APIs.

Responsibilities

Code, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes, feature stores, and vector databases in accordance with best practices and security standards.
Partner with data scientists, AI researchers, data and enterprise architects, and business stakeholders to identify, extract, clean, and format structured and unstructured data for AI/ML model training, fine-tuning, and feature extraction.
Lead evaluation, research, and experimentation with batch and streaming data technologies, LLM data preparation frameworks, and MLOps tools.
Showcase emerging AI and data technologies as a technical lead and enable adoption of modern data techniques across the organization.
Contribute to defining and refining processes and procedures for the data engineering practice.
Educate and develop ETL developers on cloud-based data engineering initiatives to support their transition into data engineering.
Assure corporate data integrity and accuracy, with particular attention to data security.
Ensure high data quality for Data Services, Analytics, and Master Data Management.
Coordinate technical solutions and take responsibility for solution design, development, testing, and delivery.
Build automated, scalable, test-driven data pipelines.
Use software development practices, including Git version control, CI/CD, and release management, to enhance existing AWS CI/CD pipelines.
Collaborate with Data Engineers, DevOps engineers, and architects on improvements to DataOps tools and frameworks.