Data Infra - Telemetry and Observability (IC)

Matter Intelligence • San Francisco, CA

7d • Onsite

About The Position

Matter is hiring a Telemetry and Observability Engineer to make datasets, training runs, models, environments, agents, and missions observable as one AI system. Reporting to Ignacio Cases Martin, this individual contributor will connect flight and mission reality with infrastructure, model, and product behavior through shared telemetry, traces, alerts, reliability mechanisms, and operational context.

Requirements

• Experience with ML observability, telemetry, time-series systems, distributed training or inference, cloud infrastructure, or production AI platforms.
• Strong hands-on engineering skills in observability, automation, APIs, infrastructure, incident tooling, and systems debugging.
• Understanding of metrics, logs, traces, events, model and data monitoring, service objectives, capacity, release safety, and incident diagnosis.
• Ability to reason about AI failure modes including bad datasets, training instability, stale checkpoints, drift, version mismatch, environment bugs, and agent-tool failure.
• Experience designing telemetry schemas and correlation workflows across multiple systems, time domains, and ownership boundaries.

Nice To Haves

• Experience with MLOps, GPU workloads, model serving, agent observability, reinforcement-learning environments, mission telemetry, or scientific instrumentation.
• Experience operating high-throughput inference, streaming systems, geospatial pipelines, or mixed cloud and edge deployments.
• Experience with reliability platforms or observability products used by multiple engineering teams.
• Familiarity with aerospace, defense, regulated operations, or other settings requiring formal change control and incident evidence.

Responsibilities

• Build a common telemetry model linking mission state, datasets, code, checkpoints, accelerators, model and prompt versions, environment episodes, agent steps, tools, decisions, cost, latency, and feedback.
• Instrument distributed training, data loading, GPU utilization, checkpointing, experiment health, batch and online inference, and model-serving behavior.
• Instrument agent workflows across prompts, context, retrieval, memory, planning, tool calls, graph state, human intervention, evidence, outcomes, and safety controls.
• Connect planned-versus-actual mission state, hardware context, time, geometry, and geolocation to relevant datasets and model results.
• Establish actionable service objectives, alerts, incident interfaces, rollout and rollback signals, and escalation paths across platform, model, environment, and agent failures.
• Build capacity and cost visibility across storage, processing, accelerators, training, evaluation, inference, retrieval, and agent execution.

Benefits

• Competitive compensation based on experience
• Early-stage equity package
• 100% employer-paid health, dental, and vision coverage
• Opportunity to work on novel sensing, data, and AI systems with real-world deployment paths to the largest industries in the world

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed