Python Data Engineer

Virtusa • Tampa, FL

2d

About The Position

We are seeking a highly experienced and motivated Senior Data Engineer to join our team. The ideal candidate will have extensive experience in designing, building, and optimizing highly scalable and robust ETL/ELT pipelines. This role will be critical in shaping our data architecture, implementing Lakehouse solutions, and working with cutting-edge technologies like LLMs and Vector Search.

Requirements

• Extensive experience in designing, building, and optimizing highly scalable and robust ETL/ELT pipelines.
• Experience with PySpark and Python for data pipeline development.
• Experience with the Databricks platform.
• Knowledge of data modeling standards, specifically the Medallion architecture (Bronze, Silver, Gold layers).
• Experience with Lakehouse capabilities, including AI/Machine Learning features and Vector Search.
• Experience evaluating and integrating Large Language Model (LLM) frameworks.
• Ability to collaborate with data scientists and business stakeholders.
• Understanding of data quality, reliability, and security principles.
• Mentorship experience.

Responsibilities

• Design, develop, and maintain robust and scalable data pipelines using PySpark and Python.
• Implement and manage data solutions within the Databricks platform.
• Define and enforce data modeling standards, specifically utilizing the Medallion architecture (Bronze, Silver, Gold layers).
• Architect and implement Lakehouse capabilities, including AI/Machine Learning features and Vector Search for advanced data retrieval.
• Evaluate, integrate, and work with Large Language Model (LLM) frameworks.
• Collaborate with data scientists and business stakeholders to understand data requirements and translate them into technical solutions.
• Ensure data quality, reliability, and security throughout the data lifecycle.
• Mentor junior engineers and contribute to best practices in data engineering.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed