Remote | Big Data Engineer — $30–$80/hour

24-Mag•New York, NY

21h•Remote

About The Position

We are sharing a specialised part-time consulting opportunity for experienced Big Data Engineers to contribute technical expertise to an advanced AI development project involving large-scale data pipelines, distributed systems, databases, and modern data-engineering workflows. Selected professionals will apply practical big-data engineering experience to complex data problems involving pipeline architecture, integration, transformation, distributed processing, storage, and system reliability. The work requires strong Python skills, deep knowledge of database technologies, and the ability to design and evaluate scalable data solutions across modern engineering environments.

Requirements

Proven professional experience in big-data engineering with hands-on responsibility for large-scale data pipelines and architectures
Advanced proficiency in Python for data processing, automation, integration, and related engineering workflows
Deep understanding of relational and NoSQL databases, including optimisation, management, and performance considerations
Practical experience with distributed data-processing frameworks such as Hadoop, Spark, Flink, or comparable technologies
Strong foundation in data modelling, ETL processes, data integration, and data-warehousing principles
Ability to troubleshoot distributed systems, database performance, pipeline failures, and data-quality issues
Excellent written and verbal communication skills
Ability to explain technical concepts clearly to both engineering and non-technical stakeholders
Strong attention to detail, initiative, and ability to work independently in remote environments

Nice To Haves

Experience supporting fast-paced, startup-like, or globally distributed teams is advantageous
Familiarity with cloud-based data platforms such as AWS, GCP, or Azure is advantageous
Exposure to machine-learning operations, data-science workflows, or adjacent AI infrastructure is beneficial

Responsibilities

Design, build, and maintain scalable data pipelines and big-data architectures
Develop reliable workflows for ingesting, transforming, processing, and delivering large volumes of data
Translate technical and business requirements into robust data-engineering solutions
Apply appropriate architectural patterns for distributed and high-volume data systems
Ensure data workflows remain scalable, maintainable, and suitable for production-oriented environments
Develop data-processing, automation, and integration solutions using Python
Implement ETL and data-transformation workflows across structured and unstructured data sources
Work with distributed processing technologies and modern big-data frameworks
Build reproducible and efficient programmatic workflows for large-scale data operations
Identify opportunities to improve processing efficiency, reliability, and maintainability
Develop, manage, and optimise relational, NoSQL, distributed database, and storage systems
Apply data-modelling and data-warehousing principles to support scalable analytical and operational workloads
Monitor data-system performance and identify bottlenecks, failures, and reliability issues
Troubleshoot performance, availability, and integration problems across distributed data environments
Optimise storage, queries, and processing workflows for efficiency and resilience
Apply appropriate data-quality, security, and governance standards across engineering workflows
Collaborate with technical and cross-functional stakeholders to understand data requirements and delivery priorities
Document architectures, data flows, implementation decisions, and technical trade-offs
Communicate complex engineering concepts clearly to both technical and non-technical audiences
Support consistent engineering standards across remote and distributed project environments