Embedded Validation & Tools Engineer - OTA

ALSO • Palo Alto, CA

41d • $140,000 - $170,000

About The Position

As an Embedded OTA Validation & Tools Engineer, you will own validation, automation, and operational readiness for firmware/software update delivery across our fleet of embedded devices. You will design and run validation test programs that ensure OTA updates are robust, secure, and reliable under real-world conditions — including intermittent networks, power faults, device restarts, and edge cases that matter to customers. You’ll work closely with Firmware, Release, QA, and Cloud teams to close the loop from release to fleet health.

Requirements

• Bachelor’s degree in Electrical Engineering, Computer Science, or related field (or equivalent experience).
• 5+ years of hands-on experience validating embedded devices and firmware updates (OTA or similar).
• Strong programming/scripting ability in Python and familiarity with C/C++ for embedded debugging.
• Deep understanding of embedded platforms (embedded Linux and/or RTOS), bootloaders (e.g., U-Boot), partition schemes, and A/B update models.
• Practical experience with building automated test harnesses and integrating tests into CI pipelines (Jenkins, GitLab CI, CircleCI, etc.).
• Experience with hardware debugging tools and workflows: serial/UART, JTAG/SWD, logic analyzers, oscilloscopes, power cycling rigs.
• Familiarity with networking protocols used for OTA (HTTP(S), MQTT, CoAP) and wireless technologies (Wi-Fi, Ethernet, cellular, BLE).
• Experience analyzing logs, packet captures, and device telemetry to diagnose update failures.
• Strong debugging skills and methodical problem solving; able to reproduce complex failure modes and propose design fixes.
• Excellent communication skills — able to write clear test plans, runbooks, and postmortems.

Nice To Haves

• Experience with cloud OTA platforms and services: AWS IoT Device Management, Mender, Azure IoT, Google Cloud IoT, or custom fleet management systems.
• Knowledge of secure update practices: code signing, public key infrastructure (PKI), TPM/secure element integration, secure boot implementations.
• Experience with CAN/LIN, automotive-grade validation, or industrial protocols for fleeted devices.
• Familiarity with containerization/virtualization for test environments (Docker, QEMU).
• Experience with large-scale rollout strategies (canary/capacity ramping, staged rollouts, feature flags).
• Firmware build systems and OS build frameworks (Yocto/OpenEmbedded, Buildroot).
• Prior experience with telemetry frameworks and analytics (Prometheus, Grafana, ELK).

Responsibilities

• Design, implement and execute comprehensive OTA validation plans across device families and release types (full OS images, firmware, bootloader, A/B updates, delta/patch updates).
• Build, manage and scale automated testbeds and harnesses (hardware-in-the-loop, virtualized devices, emulators) to run regression, integration, and stress tests for OTA flows.
• Implement automation for end-to-end OTA pipelines (from build artifact signing and staging to rollout and post-update verification) using Python, shell, and CI tools.
• Validate update mechanisms and components: bootloader, update agent, package formats, rollbacks, A/B partitions, and recovery modes.
• Test update resilience under adverse conditions: intermittent networks (cellular/Wi-Fi), high latency/packet loss, power interruptions, low-battery scenarios, storage constraints, and corrupted images.
• Validate security properties: signing/verification, secure transport (TLS), secure boot, key management, and vulnerability surface introduced by update components.
• Define metrics and KPIs (success rate, rollback rate, mean time to recover, time-to-update), instrument devices and cloud backends to measure rollout health, and produce reports for release readiness.
• Perform root cause analysis on failed updates, reproduce field issues, and drive remediation with firmware and backend engineering teams.
• Maintain device lab hardware (JTAG, serial consoles, network emulators, power controllers, logic analyzers) and tooling stacks (Wireshark, serial monitors, provisioning fixtures).
• Contribute to OTA process improvements: staging strategies, canary rollouts, rollback thresholds, risk assessment templates, and runbook documentation.

Benefits

• Robust health coverage — excellent health, dental, and vision insurance covered up to 100% by ALSO, with FSA & HSA options
• One Medical membership and dedicated insurance advocates
• Rich fertility and family-building benefits with Progyny
• Flexible time off
• 401(k) match

What This Job Offers

• Job Type: Full-time
• Career Level: Senior