Staff Engineer, Data Platform (R5659)

Shield AI•San Diego, CA

10d

About The Position

Shield AI is seeking a Staff Data Platform Engineer to define and build the data foundation of the AI Factory. The Data Platform will provide a unifying, knowledge-graph-centered API layer for human and agentic workflows, connecting various data types and systems through stable identities and typed relationships. This role is a hands-on technical leadership position responsible for designing platform architecture, implementing production software, evaluating technologies, establishing data-modeling patterns, and collaborating with teams that collect and consume mission-critical data. The goal is to balance developer productivity, semantic clarity, operational reliability, system performance, portability, and long-term maintainability.

Requirements

Significant experience designing and operating distributed data solutions, storage systems, or data-intensive backend services.
Strong software engineering skills and a record of delivering production systems in languages such as Go and Python.
Deep understanding of data modeling, API design, schema evolution, identity, consistency, indexing, query planning, and data lifecycle concerns.
Experience working across multiple storage modalities, such as relational or graph databases, object storage, analytical or columnar systems, and file storage.
Experience designing reliable ingestion and access paths for high-volume or operationally important data.
Strong understanding of Kubernetes, Linux, networking, security, storage, observability, and distributed-systems fundamentals.
Experience deploying data infrastructure across cloud or customer-managed environments using modern Infrastructure as Code and platform engineering practices.
Ability to evaluate technologies through prototypes, benchmarks, operational requirements, and total lifecycle cost rather than feature lists alone.
Experience defining architecture and technical standards while remaining hands-on in implementation and debugging.
Demonstrated ability to collaborate with ML researchers, autonomy engineers, test teams, platform engineers, and product stakeholders.
Clear technical communication and the ability to make complex data architecture understandable to both specialists and downstream users.

Nice To Haves

Experience with specialized modern databases (e.g., graph, OLAP, etc.)
Graph-backed retrieval, agent tooling, structured RAG, provenance-aware context construction, or explainable retrieval systems.
S3-compatible APIs, cloud object storage, content-addressable storage, multipart transfer, or large-file lifecycle management.
Apache Arrow, Parquet, columnar formats, time-series data, or high-performance analytical query systems.
OpenAPI, AsyncAPI, WebSockets, generated SDKs, and long-lived public API contracts.
Kubernetes storage and data operators, Terraform, Helm, GitOps, and repeatable platform distribution.
Distributed execution technologies such as Ray and experience connecting workflow execution to data lineage and artifact management.
Streaming and ingestion technologies such as Kafka, NATS, Redpanda, or comparable event-driven systems.
Data and analysis in a robotics or AI domain.
ML data lifecycle systems, experiment tracking, dataset management, evaluation infrastructure, feature or artifact stores, and model versioning.
Observability tools, distributed tracing, and benchmarking.
Data security, authorization, governance, retention, classification, and auditability across shared platforms.

Responsibilities

Lead the architecture and implementation of the knowledge graph and multi-modal API layer.
Research, optimize, and maintain the storage, indexing, query, ingestion, and compute infrastructure.
Establish durable, best-practice patterns for schema modeling, relationships, lineage, and schema evolution.
Build APIs that enable agents to retrieve structured, connected, and explainable context.
Establish recommended storage and compute profiles, deployment patterns, benchmarks, and operational guidance.
Partner directly with autonomy, ML, test, infrastructure, product, and customer-facing teams to turn real workflows into reusable platform capabilities.
Deliver integrations that make important data easy to collect and aggregate.
Create self-service APIs, SDKs, tools, examples, and diagnostics to improve developer experience.
Evaluate emerging data and AI infrastructure technologies and guide implementation across team boundaries.
Establish expectations for observability, performance, reliability, security, data integrity, disaster recovery, and lifecycle management.