Senior Software Engineer – Network Observability

Clockwork.io • Palo Alto, CA

2d • $140,000 - $210,000

About The Position

Clockwork Systems is seeking an experienced Senior Software Engineer to contribute to the architecture, development, and scaling of a high-performance network monitoring and observability platform. The role focuses on building systems that provide deep visibility into RDMA, RoCE, InfiniBand, and TCP/IP networks, and calls for experience in distributed systems, Linux networking, and modern observability stacks.

The engineer will build network monitoring and telemetry capabilities, including dashboards, alerts, diagnostics, anomaly detection, SLA monitoring, and traffic analysis workflows. They will investigate production issues across application, operating system, server, RDMA, and network layers, with attention to low-latency data collection, aggregation, and alerting. The role also involves cross-functional collaboration, improving engineering practices, automating operational workflows, and contributing to the platform’s technical direction.

Clockwork Systems develops software-driven AI fabrics to improve GPU cluster utilization through cross-stack observability, workload fault tolerance, and performance acceleration. The position offers a stated base salary range of $140,000–$210,000, with potential stock options or other equity awards subject to the company’s equity program, approvals, and vesting schedules. The company also mentions competitive compensation, a benefits package, and catered lunch.

Requirements

• Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.
• Strong hands-on programming experience in C++, Go, Python, Rust, or similar systems programming languages.
• Proven experience delivering complex infrastructure projects and contributing to the design and implementation of distributed systems.
• Experience building distributed systems, backend services, telemetry pipelines, or observability platforms.
• Hands-on experience with RDMA, RoCE, InfiniBand, or other high-performance network fabrics.
• Familiarity with libibverbs, RDMA verbs, RDMA CM, queue pairs, completion queues, memory registration, and related RDMA concepts.
• Strong knowledge of Linux networking, TCP/IP, DNS, HTTP, routing, MTU, congestion control, packet loss, latency, and performance tuning.
• Experience with traceroute-style diagnostics, path discovery, network reachability checks, synthetic probes, or active network measurements.
• Experience with monitoring and visualization platforms such as Prometheus, Grafana, Datadog, Splunk, OpenTelemetry, or similar tools.
• Strong debugging skills across software, operating system, server, and network layers.
• Experience operating production systems in Linux-based environments.
• Strong technical judgment and ability to build reliable, scalable, and maintainable systems.

Nice To Haves

• Experience supporting AI/ML, HPC, storage, or GPU cluster infrastructure workloads.
• Experience with large-scale RoCE or InfiniBand deployments.
• Experience with NCCL, distributed training infrastructure, or AI cluster diagnostics.
• Experience with eBPF, XDP, DPDK, perf, tcpdump, Wireshark, ethtool, iproute2, rdma-core, or Linux kernel networking tools.
• Experience with cloud infrastructure on AWS, GCP, or Azure.
• Experience with Kubernetes, service discovery, configuration management, and infrastructure automation.
• Knowledge of security, compliance, and infrastructure best practices.
• Experience designing time-series data systems, alerting pipelines, or high-cardinality telemetry platforms.

Responsibilities

• Design, develop, and scale high-performance network monitoring platforms for RDMA, RoCE, InfiniBand, and TCP/IP infrastructure.
• Build backend telemetry services, observability dashboards, alerts, diagnostics, anomaly detection, SLA monitoring, and traffic analysis workflows.
• Troubleshoot complex production issues across application, operating system, server, RDMA, and network layers.
• Optimize low-latency collection, aggregation, and alerting.
• Collaborate with cross-functional teams to deliver scalable solutions.
• Improve engineering practices and automate operational workflows.
• Contribute to the technical direction of the platform.

Benefits

• Competitive compensation.
• A great benefits package.
• Catered lunch.
• Eligible to participate in the company’s equity program, which may include stock options or other equity awards, subject to applicable terms and approvals.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior