Principal Linux Engineer

MESO SCALE DIAGNOSTICS, LLC.•Rockville, MD

2d•$116,600 - $177,800•Onsite

About The Position

The Principal Linux Engineer is the senior technical authority for the enterprise Linux platform and serves as the Linux support lead. The role owns the design, build standards, hardening, and day-to-day reliability of the Linux server estate. It also supports Debian 13 ("trixie") on ARM hardware embedded in company products, partnering with Product Engineering and R&D on OS image builds, hardening, patching, and field escalations. The engineer establishes engineering, scripting, and automation standards across both environments and is the final escalation point for complex Linux incidents and root-cause analysis. The position manages the Linux support queue and on-call rotation, mentors junior and mid-level engineers, and partners with Security, Network, Database, Applications, and Quality teams to keep systems available, secure, compliant, and audit-ready in a 24x7x365 environment. The work takes place in a traditional office environment, with some local travel between office sites, and may require working in data centers, data closets, and other secure environments. The annual base salary range is $116,600 to $177,800. Employees may qualify for a discretionary or non-discretionary bonus. Benefits include medical, dental, vision, and prescription coverage; a 401(k) plan with company matching; flexible spending accounts; company-paid short- and long-term disability insurance; group life and accidental death and dismemberment insurance; paid vacation, sick leave, and holidays; paid parental leave; and an employee assistance program. Additional voluntary perks include a fitness club membership contribution, pet insurance, identity theft protection, home and auto insurance discounts, and optional supplemental life insurance.

Requirements

Bachelor's degree in Computer Science, Information Systems, or equivalent work experience.
At least 10 years of progressive Linux systems administration or engineering experience in production data center environments.
At least 3 years in a lead, principal, or equivalent senior technical capacity.
Experience building, hardening, and supporting an embedded or appliance Linux OS shipped as part of a product, including reproducible image creation, first-boot provisioning, and read-only or A/B root filesystem strategies.
Hands-on experience supporting databases on Linux, including Oracle, MySQL/MariaDB, PostgreSQL, and MongoDB.
Experience in production data center environments, including clustering, high availability, and disaster recovery for Linux workloads.
Experience in a regulated environment such as GxP or ISO, including change control, documentation, and audit support.
Expert-level knowledge of enterprise Linux administration and internals, including RHEL, CentOS/Rocky/AlmaLinux, Debian (including Debian 13 "trixie"), and Ubuntu.
Knowledge of kernel tuning, systemd, LVM, filesystems, networking, and performance diagnostics.
Experience with Linux security hardening and tuning, CIS benchmarks, SELinux, and vulnerability remediation.
Expert knowledge of installation and configuration procedures for physical and virtual servers.
Expert knowledge of patch and errata management at scale and backup and restore operations.
Expert knowledge of configuration management and automation at scale; Ansible is required, with Terraform, Git-based version control, and CI/CD pipelines.
Hands-on Debian expertise, specifically Debian 13 on Raspberry Pi, including APT and dpkg, .deb packaging, apt pinning and preferences, unattended-upgrades, and local/private APT repository management.
Working knowledge of ARM64/aarch64 Linux and Raspberry Pi platform specifics, including boot chain and firmware, device tree overlays, GPIO/I2C/SPI and serial peripherals, SD/eMMC endurance, and thermal and power constraints.
Familiarity with product OS release qualification, remote upgrades and patch delivery to fielded devices, rollback strategy, and CVE tracking against a shipped software bill of materials.
Expertise in Shell and Python scripting for automation, tooling, and diagnostics.
Strong understanding of DNS, HTTP/S, LDAP, SSH, NFS, Kerberos, SMTP, and SNMP, including Active Directory integration of Linux hosts.
Understanding of basic network concepts and the OSI model.
Familiarity with web technologies, web applications, web services, service-oriented architectures, and network/web-related protocols.
Ability to provide day-to-day guidance and mentoring in a supportive, collaborative manner.
Advanced problem-solving and root-cause analysis skills, with composure while leading major incident response.
Strong verbal and written communication skills, including briefing senior leadership on risk, incidents, and roadmap.
Ability to move or lift up to 50 pounds.
Ability to work standing up in data centers, data closets, and other secure environments, and to lift moderately heavy equipment when required.

Nice To Haves

AWS experience; experience with cloud hosting, SaaS, or PaaS technologies is requested, with AWS preferred.
Red Hat certification (RHCSA/RHCE) is strongly preferred.
Cloud, security, or ITIL certifications are a plus.
Experience with VMware vSphere and container platforms such as Docker and Kubernetes/OpenShift is preferred.
Perl scripting is a plus.

Responsibilities

Serve as technical lead and final escalation point for Linux support, owning triage, root-cause analysis, and permanent corrective action for complex incidents.
Own the architecture, build standards, and lifecycle of the Linux estate, including capacity planning and hardware and OS refresh.
Define and enforce configuration management and infrastructure-as-code practices using Ansible, Terraform, Git, and CI/CD pipelines.
Provide engineering and support ownership for the Debian Linux OS distributed with company products, including image definition, kernel and device-tree configuration, package selection, hardening, and release qualification.
Partner with Product Engineering, R&D, Manufacturing, and Field Service on Raspberry Pi image provisioning, unattended upgrade and patch delivery mechanisms, and escalated field issues.
Lead performance analysis, fault isolation, and reporting of service exceptions across the Linux estate.
Partner with Network, Security, Windows, Database, and Application teams on data center builds, migrations, and cross-platform projects.
Mentor and develop Linux engineers and administrators, providing technical direction, training, and workload prioritization.
Author and maintain SOPs, runbooks, and change/configuration records in line with company quality and regulatory requirements.
Perform other duties as assigned by the organization.