Senior Databricks Engineer

Infosys Pontoon • Boston, MA

14h • Onsite

About The Position

The Senior Databricks Engineer will develop data engineering and data-processing solutions using Databricks and cloud services, primarily Azure, for data warehouse projects. The role requires extensive hands-on experience with Databricks, Hadoop, Python, PySpark, Scala, SQL, and Azure or AWS cloud services, including developing and integrating cloud-based solutions and optimizing processing of large datasets.

The engineer will work with business stakeholders, client architects, and design teams to gather requirements, understand business needs and architecture, and contribute to design and development. The position also involves data ingestion and orchestration, DevOps and CI/CD practices, and collaboration in an onshore/offshore Agile environment. Financial industry experience is required. Databricks and Azure certifications are an advantage.

Requirements

• At least 9 years of total experience, including 5 years of relevant experience in the mandatory skills.
• Hands-on Databricks experience with Azure cloud services in data engineering or data-processing development projects involving data warehouses.
• Experience in Azure- or AWS-based cloud development and integration.
• Experience with Databricks, Hadoop, Python, PySpark, Scala, SQL, and Azure/AWS cloud services.
• Proficiency in Scala and Spark DataFrames for data processing and application development.
• Knowledge and experience with facts, dimensions, data partitions, and large-volume data.
• Experience handling large datasets in Databricks and optimizing performance.
• Practical experience in data engineering, data ingestion and orchestration, and DevOps with CI/CD tools.
• Knowledge of Spark and Databricks SQL for data engineering pipelines in a data warehouse.
• Experience in an onshore/offshore model and Agile methodology.
• Ability to gather requirements, understand business needs, and work with technical stakeholders on design and development.
• Experience working with client architects and design teams to understand architecture and requirements and perform development.
• Experience working in the financial industry.

Nice To Haves

• Databricks certification.
• Azure certification.

Responsibilities

• Develop data engineering and data-processing solutions with Databricks for data warehouse projects.
• Build and integrate cloud-based solutions using Azure and/or AWS.
• Develop data processing and applications using Scala, Spark DataFrames, and related tools.
• Work with large datasets and optimize Databricks performance.
• Implement data engineering pipelines, data ingestion, and orchestration, with accompanying DevOps and CI/CD practices.
• Gather requirements, understand business needs, and discuss design and development activities with technical stakeholders.
• Work with client architecture and design teams to understand architecture and requirements and carry out development.
• Collaborate in an onshore/offshore model and Agile methodology.

What This Job Offers

• Career Level: Senior
• Education Level: No Education Listed