Senior Data Engineer (Azure, Databricks)

OPTIVEUM sp. z o.o.•Swoyersville, PA

1d•Onsite

About The Position

The Senior Data Engineer will join an innovative data team at the financial services (captive finance) arm of a global automotive company. Optiveum is the recruitment and contracting party, and the successful candidate will sign the cooperation agreement directly with Optiveum. The role focuses on building and optimising ETL pipelines in the Azure and Databricks ecosystem, migrating data from legacy systems to next-generation platforms, ensuring data quality, and collaborating with cross-functional teams to deliver actionable insights. The position is full-time (40 hours per week), onsite in Wrocław, under a B2B contract. English proficiency at C1 level is required. The rate is up to 165 PLN per hour net plus VAT.

Requirements

At least 5 years of experience in data engineering or a related field, with a strong focus on ETL and cloud data tools.
Bachelor's degree in Computer Science, Engineering, or a related field; a Master's degree is preferred.
Microsoft Certified Azure Data Engineer certification.
Databricks Certified Data Engineer certification.
Proficiency in Python, Java, or Scala.
Hands-on experience with MS Cloud SQL Server, Azure Data Factory, Azure Databricks, PySpark, Spark SQL, ADLS, Power BI, SSIS, and SSRS.
Strong SQL skills and experience with relational databases.
Familiarity with big data technologies and data processing frameworks.
Experience with Agile/Scrum.
Strong problem-solving and communication skills, and a collaborative mindset.
English proficiency at C1 level.

Nice To Haves

Experience in financial services, ideally in a captive finance company, including customer, dealer, and supplier data.
Experience delivering global data marts used across regions.
Java.

Responsibilities

Extract, combine, transform, load, and cleanse data from various sources to deliver high-quality datasets for analysis, including migrating data from legacy systems.
Work with data modellers and business analysts to understand data models and source-to-staging mappings.
Create analysis-ready data structures, including DDLs for MS Cloud SQL Server tables and views in the Pond database, a subset of the Data Lake.
Design and develop Databricks notebooks to extract data from input files and join related entities into Shared File Formats (SFFs).
Configure Ingestion Framework tables to load staging and Data Lake Core (DLC) tables.
Build large-scale batch and real-time data pipelines in cloud or on-premises environments, and develop performance-efficient algorithms.
Apply CI/CD best practices and introduce automation in existing and future integration flows.
Track data consumption patterns and work with data governance and information stewards on the curated data catalogue.
Drive improvements in data acquisition, testing, and tooling to raise data quality.
Monitor performance and optimise SQL queries.
Prepare detailed design documents based on business and system requirement documents.
Work in cross-functional agile teams and participate in requirement analysis and estimates.