Data Platform Engineer

BrightAI Corporation • Palo Alto, CA

1d

About The Position

As a Data Platform Engineer at a high-growth company integrating AI, IoT, and cloud-native services, you will build and maintain the data infrastructure that powers its products, services, and insights. You will join a multidisciplinary team that ingests, processes, and manages large streams of sensor and operational data from devices including drones, robots, industrial systems, and smart environments.

The role involves developing scalable data ingestion and storage systems, internal tools, and end-to-end workflows using Python, Golang, SQL, AWS, and Terraform. You will collaborate with Cloud, Edge, Product, and AI teams, monitor and optimize pipelines and datasets, and contribute to architecture reviews and platform security, compliance, and best practices.

Requirements

• At least 5 years of professional experience in software engineering or data engineering.
• Strong programming skills in Python and Golang.
• Deep understanding of SQL and modern data lake architectures, such as Parquet, Iceberg, or Delta Lake.
• Hands-on experience with AWS services, including S3, Lambda, Glue, Kinesis, Athena, and RDS.
• Proficiency with Terraform for infrastructure deployment and management.
• Experience with large-scale real-time or batch data ingestion and designing fault-tolerant ETL/ELT pipelines.
• Familiarity with event-driven architectures and messaging systems such as Kafka or Kinesis.
• Strong debugging and optimization skills across cloud, network, and application layers.
• Excellent collaboration, communication, and documentation skills.

Nice To Haves

• Experience with time-series or IoT sensor data at industrial scale.
• Familiarity with analytics tools and data warehouse integration, such as Redshift or Snowflake.
• Exposure to gRPC and protobuf-based data contracts.
• Experience supporting ML pipelines and feature stores.
• Working knowledge of Kubernetes concepts.
• Prior startup experience or comfort working in fast-paced, iterative environments.

Responsibilities

• Design, build, and maintain scalable, reliable, high-throughput ingestion pipelines for structured and semi-structured data.
• Implement secure data lake and SQL-based storage architectures optimized for performance and cost.
• Develop and maintain internal data ingestion tools and frameworks using Python, Golang, and SQL.
• Collaborate with Cloud, Edge, Product, and AI teams to define data contracts, schemas, and retention policies.
• Use AWS infrastructure, including Argo Workflows, S3, Lambda, Glue, Kinesis, Athena, and RDS, to support end-to-end data workflows.
• Manage data platform infrastructure using Terraform and Infrastructure-as-Code practices.
• Monitor data pipeline quality, latency, and failures using tools such as CloudWatch, SumoLogic, or DataDog.
• Optimize storage, partitioning, and query performance across large-scale datasets.
• Participate in architecture reviews and ensure the platform follows security, compliance, and best-practice standards.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed