Staff Engineer, Data Platform (R5659)

Shield AI•San Diego, CA

18h•$150,000 - $230,000

About The Position

We are looking for a Staff Data Platform Engineer to help define and build the data foundation of the AI Factory. The Data Platform provides a unifying, knowledge-graph-centered API layer for human and agentic workflows. It connects configurations, requirements, software versions, test executions, files, signals, training data, and results through stable identities and typed relationships. It also provides consistent access to the storage and compute systems behind those data products. This is a hands-on technical leadership role. You will design platform architecture, implement production software, evaluate storage and compute technologies, establish data-modeling patterns, and work directly with teams collecting and consuming mission-critical data. Success requires balancing developer productivity, semantic clarity, operational reliability, system performance, portability, and long-term maintainability.

Requirements

Significant experience designing and operating distributed data solutions, storage systems, or data-intensive backend services.
Strong software engineering skills and a record of delivering production systems in languages such as Go and Python.
Deep understanding of data modeling, API design, schema evolution, identity, consistency, indexing, query planning, and data lifecycle concerns.
Experience working across multiple storage modalities, such as relational or graph databases, object storage, analytical or columnar systems, and file storage.
Experience designing reliable ingestion and access paths for high-volume or operationally important data.
Strong understanding of Kubernetes, Linux, networking, security, storage, observability, and distributed-systems fundamentals.
Experience deploying data infrastructure across cloud or customer-managed environments using modern Infrastructure as Code and platform engineering practices.
Ability to evaluate technologies through prototypes, benchmarks, operational requirements, and total lifecycle cost rather than feature lists alone.
Experience defining architecture and technical standards while remaining hands-on in implementation and debugging.
Demonstrated ability to collaborate with ML researchers, autonomy engineers, test teams, platform engineers, and product stakeholders.
Clear technical communication and the ability to make complex data architecture understandable to both specialists and downstream users.

Nice To Haves

Experience with specialized modern databases (eg graph, OLAP, etc...)
Graph-backed retrieval, agent tooling, structured RAG, provenance-aware context construction, or explainable retrieval systems.
S3-compatible APIs, cloud object storage, content-addressable storage, multipart transfer, or large-file lifecycle management.
Apache Arrow, Parquet, columnar formats, time-series data, or high-performance analytical query systems.
OpenAPI, AsyncAPI, WebSockets, generated SDKs, and long-lived public API contracts.
Kubernetes storage and data operators, Terraform, Helm, GitOps, and repeatable platform distribution.
Distributed execution technologies such as Ray and experience connecting workflow execution to data lineage and artifact management.
Streaming and ingestion technologies such as Kafka, NATS, Redpanda, or comparable event-driven systems.
Data and analysis in a robotics or AI domain.
ML data lifecycle systems, experiment tracking, dataset management, evaluation infrastructure, feature or artifact stores, and model versioning.
Observability tools, distributed tracing, and benchmarking.
Data security, authorization, governance, retention, classification, and auditability across shared platforms.

Responsibilities

Develop a unifying Graph API: Lead the architecture and implementation of the knowledge graph and multi-modal API layer that serves as the backbone for human, service, and agentic workflows.
Own DataOps infrastructure: Research, optimize, and maintain the storage, indexing, query, ingestion, and compute infrastructure used throughout the data lifecycle.
Establish best-practices: Establish durable, best-practice patterns for schema modeling, relationships, lineage, and schema evolution.
Turbocharge agentic data access: Build APIs that enable agents to retrieve structured, connected, and explainable context rather than relying only on keyword or vector similarity.
Develop reference architectures: Establish recommended storage and compute profiles, deployment patterns, benchmarks, and operational guidance for both internal and customer-managed infrastructure.
Advise downstream teams: Partner directly with autonomy, ML, test, infrastructure, product, and customer-facing teams to turn real workflows into reusable platform capabilities from modeling to integrations.
Build first-party integrations: Deliver integrations that make important data easy to collect and aggregate, including data produced by simulations, test infrastructure, training systems, and edge devices.
Improve developer experience: Create self-service APIs, SDKs, tools, examples, and diagnostics that make correct data modeling and ingestion the easiest path.
Drive technical direction: Evaluate emerging data and AI infrastructure technologies, make principled build-versus-buy decisions, and guide implementation across team boundaries.
Raise operational quality: Establish expectations for observability, performance, reliability, security, data integrity, disaster recovery, and lifecycle management.