Big Data Specialist (Multiple Positions) | ZR_925_JOB
About The Position
Agilisium Consulting LLC is hiring Big Data Specialists to develop and optimize large-scale data pipelines and data processing systems. The role involves building Databricks notebooks for SnapLogic pipeline data cleansing and processing, managing data manipulation and orchestration, and designing standardized, modular pipelines that meet client performance expectations. The specialist will develop validation processes to protect data integrity, establish scalable and consistent frameworks for multiple CDL layers, create reusable code modules, and maintain version control practices. The position also leads data integration efforts and develops automated ingestion of Google Analytics data to enable actionable insights. The role requires experience with PySpark, Databricks, SQL, Amazon Redshift, SQL DW, Apache Airflow, SnapLogic, Informatica PowerCenter, AWS services, Medallion Architecture, Delta Lake, Parquet, and dimensional and relational data modeling. The job is based in Westlake Village, California, and various unanticipated locations throughout the United States. The employee must be willing to work anywhere in the U.S. and may need to relocate to client sites; relocation is paid by the employer pursuant to internal policy.
Requirements
- Bachelor’s degree or foreign equivalent in Computer Science, Computer Engineering, or a related field.
- Five years of experience in the position offered, as a Software Engineer, or in a related position.
- Five years of experience developing and optimizing data pipelines in PySpark and Databricks for large-scale data ingestion and processing, and integrating GitLab version control for code management and automated deployments.
- Five years of experience implementing advanced SQL development and optimization techniques for complex data modeling, query performance tuning, and relational warehouse management in Amazon Redshift and SQL DW.
- Five years of experience engineering and automating ETL/ELT workflows using Apache Airflow, SnapLogic, and Informatica PowerCenter for reliable, reusable, end-to-end data integration across heterogeneous systems.
- Five years of experience architecting and operating cloud-native big-data ecosystems on AWS, including AWS S3, EC2, Kinesis, and Lambda for low-latency ingestion and analytics pipelines.
- Five years of experience designing and implementing data quality and governance frameworks within the Medallion Architecture (Bronze, Silver, and Gold layers), including data reliability, lineage, and compliance with enterprise governance standards.
- Five years of experience implementing Delta Lake optimization strategies, including Z-Ordering, OPTIMIZE, and VACUUM, and designing partition pruning and Parquet columnar data formats to improve performance and storage efficiency.
- Five years of experience architecting and optimizing dimensional and relational data models using Star and Snowflake schemas within Databricks and Amazon Redshift.
Responsibilities
- Develop Databricks notebooks to implement SnapLogic pipeline data cleansing and processing.
- Optimize and manage data manipulation tasks.
- Implement and automate data flow orchestration.
- Design and implement modular, standardized pipelines to meet client performance expectations.
- Develop and execute validation processes to ensure data integrity throughout the pipeline.
- Establish a standardized framework for multiple CDL layers to ensure scalability and consistency.
- Create reusable code modules and maintain version control practices.
- Lead data integration to drive system improvements.
- Develop automated processes for ingesting Google Analytics data to enable actionable insights.
Benefits
- Employer-paid relocation to client site locations pursuant to internal policy.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
