Senior Data Engineer (Databricks)

Sparq

2h • Hybrid

About The Position

We are seeking a Senior Data Engineer with Databricks expertise to join our team. This role offers the opportunity to work with cutting-edge technologies, collaborate with a diverse team, and contribute to large-scale data migrations. You will have ongoing investment in your growth through training, mentorship, and industry certifications, with hands-on exposure to technologies across multiple industries. We foster an inclusive culture that celebrates great work through recognition programs and employee awards, including our AI Innovators Program. This position offers remote or hybrid work options, a competitive salary with bonus opportunities, a robust benefits package, matching 401(k) plan, substantial PTO, and tuition reimbursement.

Requirements

• Recent hands-on experience building production batch solutions in Databricks, ideally within the past 12–18 months.
• Strong development experience with Python and Apache Spark.
• Advanced SQL skills, including complex query development and performance tuning.
• Consultative approach and problem solving skills to successfully align digital solutions with long-term business goals of the client.
• Experience delivering large-scale migrations from legacy data platforms to cloud-based environments, preferably involving Teradata or a comparable platform.
• Experience supporting legacy and cloud systems operating in parallel during a migration.
• The ability to lead design and technical-direction discussions while remaining actively involved in coding and implementation.
• Hands-on experience using AI tools to enhance daily work, or a strong desire to do so, including a willingness to experiment, learn, and champion AI adoption within your role.
• Strong communication skills across technical teams, architects, business partners, and leadership.
• Availability to participate in an on-call rotation.
• Must be authorized to work in the U.S. without sponsorship.

Nice To Haves

• Experience with Azure data services—such as Data Factory, Synapse Analytics, Event Hubs, Delta Lake, Cosmos DB, and Azure DevOps—is preferred.
• Exposure to Big Data technologies, NoSQL databases, Git, Jenkins, CI/CD practices, and Agile delivery methodologies is a plus.
• If you have experience in AI or a proven track record of rapidly learning and mentoring others on emerging tech, you’ll fit right in.

Responsibilities

• Build and migrate large-scale batch data pipelines from legacy data platforms into Databricks.
• Develop and optimize production-grade solutions using Python, Spark, and advanced SQL.
• Support parallel legacy and cloud data environments throughout the migration and validation process.
• Serve as a subject matter expert while remaining hands-on with day-to-day development and troubleshooting.
• Actively use AI tools, including Claude through our Anthropic partnership, to streamline tasks, improve the quality of your work, and share best practices with teammates while continuously seeking new ways to integrate AI into your everyday workflows.
• Lead technical design conversations and guide implementation within an established architecture and project framework.
• Collaborate with developers, architects, and business stakeholders to communicate technical decisions, risks, and progress.
• Improve pipeline reliability and performance through testing, monitoring, tuning, and production support.
• Use GitHub Copilot when helpful to support development productivity.
• Participate in a rotating on-call schedule approximately every four to six weeks.

Benefits

• Ongoing investment in your growth through training, mentorship, and industry certifications
• Hands-on exposure to cutting-edge technologies across multiple industries
• Recognition programs and employee awards within an inclusive culture that celebrates great work, including our AI Innovators Program
• Remote or hybrid work options
• Competitive salary + bonus opportunities
• Robust benefits package
• Matching 401(k) plan
• Substantial PTO
• Tuition reimbursement

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed