Data Engineer Consultant - NYC
About The Position
Indicium AI is a global AI-native consultancy trusted by leading enterprises to deliver AI into production at scale. They have proven experience across various industries and work with top partners like Databricks, AWS, OpenAI, and Microsoft. The opportunity involves developing, implementing, and maintaining scalable, reliable, and high-performance data solutions. This includes designing and building data pipelines to integrate, secure, and ensure the reliability of data from diverse sources for internal and external applications, as well as for advanced analytics and strategic decision-making.
Requirements
- At least 4 years of experience in at least one programming language (Python, Java, Ruby, JavaScript, Scala, etc.).
- Experience with version control systems such as GitHub, GitLab, Bitbucket, etc.
- Advanced knowledge of SQL.
- Advanced knowledge in DBT.
- Experience or knowledge working with Databricks.
- Knowledge of algorithms and data structures.
- Experience with technologies such as Spark, Kafka, Presto, and/or Airflow and feel confident creating aggregated datasets.
- Experience with data warehouses such as Google BigQuery, Redshift, and/or Snowflake is required.
- Experience with Infrastructure as Code (IaC) tools, such as Terraform.
- Experience with cloud infrastructure (AWS, GCP, etc.).
- Experience with cloud data processing (AWS, GCP, Azure, Snowflake, Databricks).
- Certification in public cloud platforms (AWS, Azure, Snowflake, Databricks, GPC) at the Associate level or equivalent.
- Applicants must be U.S. citizens to be eligible for this role.
Responsibilities
- Perform data ingestion/integration from various sources (relational and NoSQL databases, internal and external service APIs, files, and others) and ensure data quality and consistency.
- Implement data storage solutions (Data Warehouses, Data Lakes) and optimize the performance of data queries and processing.
- Managing data loading in distributed storage (whether relational or non-relational).
- Aggregate data using distributed tools that can handle large volumes of data.
- Design, develop, and maintain robust and efficient Electronic Logistics (ELT) pipelines using data engineering best practices.
- Ensure the entire ELT process is functioning correctly through monitoring and metrics such as SLAs.
- Monitor the execution of ELT applications hosted in the cloud and on-premises.
- Ensure data security and governance by implementing data access and quality policies.
- Provision and/or maintain the data infrastructure, ensuring scalability, availability, and security.
- Automate the provisioning and management of infrastructure using Infrastructure as Code (IaC) tools.
- Disseminate DevOps and DataOps best practices within the team.
- Collaborate with teams to deliver data solutions that meet business needs.
- Research and implement new technologies and tools to improve the efficiency and scalability of data solutions.
- Having the freedom and critical thinking skills to propose and question solutions related to data engineering.
- Documenting processes, architectures, and data solutions.
Benefits
- Highly competitive salary package along with company bonus
- Personal learning budget
- Pick your own Gear! Macbooks, PCs, Accessories!
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
