Senior Data Engineer (Customer Data Products)
About The Position
NBME is seeking a Senior Data Engineer to join a growing team of data engineers and BI developers. The role supports the modernization, expansion, and optimization of NBME’s AWS data platform, including data lakes, complex integration pipelines, and scalable solutions for analytics, AI/ML, and business decision-making. The engineer will apply AI-assisted engineering practices to improve development efficiency and share best practices across the IT organization. The position works closely with data scientists, AI researchers, architects, and business stakeholders to prepare data and deliver secure, high-quality data solutions. It also provides technical leadership in evaluating emerging technologies and advancing the organization’s data engineering practices. The role is hybrid, with occasional onsite work for collaboration or meetings; NBME is prioritizing applicants within 50 miles of its Philadelphia office. Onsite interviews and onboarding may be required, with advance notice provided where applicable.
Requirements
- Bachelor’s degree.
- At least 7 years of application development experience; internship experience does not count.
- At least 4 years of experience with big data technologies.
- At least 4 years of experience with AWS cloud computing.
- At least 4 years of application development experience using Python, SQL, Scala, or Java.
- At least 4 years of experience with distributed data or computing tools such as MapReduce, Hadoop, Hive, EMR, Kafka, Spark, or MySQL.
- At least 4 years of experience with real-time data and streaming applications.
- At least 4 years of experience implementing NoSQL technologies such as Mongo or Cassandra.
- At least 4 years of data warehousing experience with Redshift.
- At least 6 years of UNIX/Linux experience, including basic commands and shell scripting.
- At least 7 years of experience with Agile engineering practices.
- At least 7 years of SQL optimization experience.
- At least 4 years of PySpark experience.
- At least 3 years of process orchestration experience with tools such as AirFlow, KubeFlow, AWS Step Functions, or Luigi.
Nice To Haves
- Proven experience implementing Generative AI, LLM data preparation pipelines, and vector databases such as Pinecone, Milvus, or pgvector.
- Strong experience building and maintaining feature stores for machine learning models.
- Experience building highly scalable, secure, production-ready APIs and Data-as-a-Service (DaaS) platforms.
- AWS Certified Data Engineer or AWS Certified Machine Learning – Specialty certification.
- At least 3 years of machine learning experience.
- Experience building a Data-as-a-Service platform.
- Experience building APIs.
Responsibilities
- Code, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes, feature stores, and vector databases in accordance with best practices and security standards.
- Partner with data scientists, AI researchers, data and enterprise architects, and business stakeholders to identify, extract, clean, and format structured and unstructured data for AI/ML training, fine-tuning, and feature extraction.
- Lead evaluation, research, and experimentation with batch and streaming data technologies, LLM data preparation frameworks, and MLOps tools.
- Showcase emerging AI and data technologies and enable adoption of modern data techniques across the organization.
- Contribute to defining and refining processes and procedures for the data engineering practice.
- Educate and develop ETL developers on cloud-based data engineering initiatives to support their transition into data engineering.
- Protect the integrity, accuracy, security, and quality of corporate data, including data used by Data Services, Analytics, and Master Data Management.
- Coordinate technical solutions and take responsibility for solution design, development, testing, and delivery.
- Build automated, scalable, test-driven data pipelines.
- Use software development practices including Git version control, CI/CD, and release management to enhance AWS CI/CD pipelines.
- Collaborate with Data Engineers, DevOps engineers, and architects to identify improvements to DataOps tools and frameworks.
Benefits
- Healthcare, dental, prescription, and vision plans.
- 401(k) with employer match.
- Tuition reimbursement plan.
- Commuter benefit for public transit or parking.
- Competitive salary and a rewarding work environment.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
