Perception Engineer

Lightspeed•Northbrook, IL

About The Position

As a Perception Engineer, you will own how LightSpeed's robots see. You will design, calibrate, and deploy the multi-camera systems on our robot cells and end-of-arm tooling, and build the perception that guides manipulation in real time. Your camera streams also feed our teleoperation and robot learning pipelines, so the quality of what you build directly shapes how well our robots learn. This role sits at the intersection of sensor hardware, 3D vision, and deep learning, all deployed on physical robots where reliability matters.

Requirements

3+ years building and deploying perception systems on physical robots or real-time systems
Hands-on experience calibrating multi-camera systems (intrinsic, extrinsic, hand-eye) and synchronizing sensors
Experience with RGB-D and stereo cameras in real deployments
Strong computer vision fundamentals: camera geometry, multi-view geometry, and 3D reconstruction
Deep learning experience with detection, segmentation, or pose estimation using PyTorch
Strong Python and C++
Experience with OpenCV and point cloud libraries such as Open3D or PCL
Experience with ROS or ROS2
Proficiency with Linux, Docker, Git, and CI/CD workflows

Nice To Haves

MS or PhD in Computer Vision, Robotics, Computer Science, or a related field
Perception for robotic manipulation, especially dexterous or tool-based manipulation
Experience with hardware video encoding (NVENC, Jetson Multimedia API), GStreamer, or WebRTC
Experience with NVIDIA Isaac ROS / NITROS, VPI, or CV-CUDA
Experience with point cloud registration and scan-to-CAD/BIM comparison
Experience with vision foundation models (for example SAM, DINOv2, FoundationPose)
Experience with egocentric video or hand pose estimation
Experience producing sensor data for robot learning datasets (e.g., rosbag or MCAP)
Edge deployment on NVIDIA Jetson with TensorRT
Simulation and synthetic data generation with Isaac Sim, MuJoCo, or similar
Background in manufacturing, industrial automation, or construction technology

Responsibilities

Select, specify, and deploy multi-camera systems (RGB, stereo, depth) for robot cells, including wrist- and tool-mounted cameras on end-of-arm tooling and dexterous hands
Own intrinsic, extrinsic, and hand-eye calibration, and keep calibration reliable across tool changes and daily operation
Synchronize cameras with robot state and teleoperation streams
Work with mechanical engineers on camera placement for coverage, occlusion, and lighting
Estimate and track robot and end-effector pose using vision systems to verify kinematics and identify drift
Compare 3D scans against CAD or BIM models to check part placement and as-built accuracy
Localize robots and tools in the workspace using fiducials (AprilTag or ChArUco) and multi-view fusion
Build object detection, segmentation, and 6-DoF pose estimation for parts, fasteners, and tools in cluttered, confined, and poorly lit workspaces
Build point cloud pipelines (filtering, segmentation, registration such as ICP, and meshing) for manipulation and scene understanding
Track hands and objects in egocentric video of human demonstrations
Build CUDA-optimized perception pipelines: GPU preprocessing, custom CUDA kernels where needed, TensorRT inference, and minimal host–device copies
Profile and optimize end-to-end latency, from photon to action, using tools like Nsight Systems
Support Deliver calibrated, time-synchronized camera data in the formats our teleoperation recording and training pipelines consume
Partner with AI/ML engineers to define observation spaces for imitation learning and vision-language-action policies
Set perception accuracy targets tied to task success, and build benchmarks to measure them
Integrate perception with ROS2-based robot control
Harden systems for real-world conditions: lighting changes, reflective surfaces, occlusion, and vibration
Collaborate closely with robotics, mechanical, and controls engineers