Big Data Engineer
About The Position
The Big Data Engineer develops and optimizes complex data pipelines, applying machine learning engineering principles to improve efficiency and scalability. The role integrates and optimizes data and model pipelines in production, investigates data inconsistencies, and documents assumptions. It also uses experimentation, statistics, and machine learning concepts to build self-running AI systems for predictive modeling. The engineer works with data science teams to review model-ready datasets and feature documentation for completeness and accuracy, discovers and analyzes raw data sources in their business context, and tracks data lineage during exploration or root-cause analysis. The role engages internal stakeholders to understand business processes and translate requirements into analytical approaches. It also maintains model-monitoring scripts, diagnoses issues, and coordinates resolutions based on alerts.
Requirements
- At least 3 years of hands-on experience designing ETL pipelines using AWS services such as Glue and SageMaker.
- Proficiency in programming, particularly Python, including PySpark and PySQL.
- Familiarity with machine learning libraries and frameworks.
- Strong understanding of cloud technologies, including AWS and Azure.
- Experience with NoSQL databases.
Nice To Haves
- Experience with API design and development.
Responsibilities
- Develop and optimize complex data pipelines for efficiency and scalability.
- Integrate and optimize data and model pipelines in production environments.
- Diagnose data inconsistencies and document assumptions.
- Use experimental methodologies, statistics, and machine learning concepts to create self-running AI systems for predictive modeling.
- Collaborate with data science teams to review model-ready datasets and feature documentation for completeness and accuracy.
- Discover and analyze raw data sources, applying business context to support model development.
- Explore data and track data lineage during inception or root-cause analysis.
- Engage internal stakeholders to understand business processes and translate requirements into analytical approaches.
- Write and maintain model-monitoring scripts, diagnose issues, and coordinate resolutions based on alerts.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
