Senior Data Engineer

American IT Systems • Sunnyvale, CA

4d • $60 - $65 • Onsite

About The Position

We are seeking a Data Engineer with Spark & Streaming skills builds real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML.

Requirements

• Strong proficiency in Python, SQL, and potentially Scala/Java.
• Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
• Experience with messaging queues like Apache Kafka, or Pub/Sub.
• Familiarity with GCP, Azure data services.
• Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.
• Total IT Experience - Minimum 8 years
• GCP - 4 + years of recent GCP experience

Nice To Haves

• Experience with Airflow, Databricks, Docker, Kubernetes is a plus.

Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies (Kafka, Flink).
• Build and optimize scalable data architectures, including data lakes, data warehouses (Big Query), and streaming platforms.
• Performance Tuning: Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness.
• Data Quality: Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed