Eli Lilly-posted 3 months ago
$70,500 - $200,200/Yr
Full-time • Senior
Boston, MA
Chemical Manufacturing

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We're looking for people who are determined to make life better for people around the world. Tech@Lilly is seeking a highly skilled Senior Data Engineer who can implement and optimize large-scale Lakehouse solutions and drive the evolution of our modern data platform while providing technical leadership to a growing team. The ideal candidate will have hands-on experience with modern data engineering technology stack and a proven track record of managing engineering talent in fast-paced environments.

  • Design and implement comprehensive Lakehouse architecture solutions using technologies like Databricks, Snowflake, or equivalent platforms
  • Build and maintain real-time and batch data processing systems using Apache Spark, Kafka, and similar technologies
  • Architect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data
  • Develop data transformation workflows using tools like DBT, Airflow, or Databricks
  • Lead the technical strategy for data lake and data warehouse integration, ensuring optimal performance and cost efficiency
  • Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols
  • Implement centralized data catalog system and enhance data discovery using technologies like Elastic Search / Open Search
  • Establish monitoring and alerting systems for data pipeline health using technologies like Apache Superset
  • Drive adoption of modern data engineering best practices including Infrastructure as Code, CI/CD, and automated testing
  • Collaborate with data scientists, analysts, and business stakeholders to translate requirements into robust technical solutions
  • Mentor a team of 3-5 data engineers
  • Foster a collaborative team culture focused on continuous learning and innovation
  • Experience with streaming data technologies (Kafka)
  • Familiarity with data cataloging tools (Apache Atlas or DataHub)
  • Familiarity with high performance data service framework (Arrow Flight)
  • Industry certifications in cloud platforms or big data technologies
  • Expert-level proficiency in Python and SQL for data transformation and pipeline development
  • Strong experience with Apache Spark for big data processing and analytics
  • Hands-on experience with cloud platforms (AWS or Azure) and their data services
  • Proficiency with Infrastructure as Code tools (Terraform, CloudFormation)
  • Experience with containerization (Docker, Kubernetes) and orchestration platforms
  • Knowledge of data modeling techniques for both analytical and operational workloads
  • Understanding of data governance, security, and compliance requirements
  • Knowledge in the pharmaceutical or life sciences domain
  • Eligibility to participate in a company-sponsored 401(k)
  • Pension
  • Vacation benefits
  • Eligibility for medical, dental, vision and prescription drug benefits
  • Flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts)
  • Life insurance and death benefits
  • Certain time off and leave of absence benefits
  • Well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities)
© 2024 Teal Labs, Inc
Privacy PolicyTerms of Service