Tech Lead - Software Engineer

apnconsultinginc - • Santa Clara, CA

14h • Onsite

About The Position

The posting seeks an experienced Senior Databricks Engineer to design, develop, and optimize large-scale data processing solutions. The role involves building data pipelines and ETL/ELT processes on cloud-based data platforms to support analytics and business intelligence initiatives. The listed location is Santa Clara, California, with an in-person interview on the first day after the first interview round is cleared.

Requirements

• 8+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Expertise in PySpark, Python, and SQL.
• Experience with Spark optimization techniques and performance tuning.
• Strong understanding of data warehousing concepts and dimensional modeling.
• Experience building ETL/ELT pipelines.
• Knowledge of Delta Lake, Spark SQL, and Databricks workflows.
• Experience with Git, CI/CD, and Agile methodologies.
• Strong analytical and problem-solving skills.
• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• Excellent communication and stakeholder management skills.

Nice To Haves

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Knowledge of Azure Data Factory (ADF), Airflow, or other orchestration tools.
• Experience with streaming technologies such as Kafka or Structured Streaming.
• Exposure to Power BI, Tableau, or other analytics tools.
• Databricks or cloud certifications.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize ETL/ELT frameworks for large volumes of structured and unstructured data.
• Develop data ingestion solutions that move data from multiple source systems into cloud data lakes and warehouses.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
• Implement data quality, validation, and monitoring frameworks.
• Perform data modeling and transformation for reporting and analytics use cases.
• Troubleshoot production issues and provide timely resolutions.
• Follow best practices for coding, testing, version control, and deployment.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior