Principal Linux Engineer

MSD Careers•Rockville, MD

1d•$116,600 - $177,800•Onsite

About The Position

The Principal Linux Engineer is the senior technical authority for the enterprise Linux platform and serves as the Linux support lead. The role owns design, build standards, hardening, and day-to-day reliability of the Linux server estate, as well as the Debian-based operating system (Debian 13 "trixie" on ARM hardware) embedded in certain company products. The engineer partners with Product Engineering and R&D on OS image builds, hardening, patching, and field escalations; sets engineering, scripting, and automation standards across both estates; and serves as the final escalation point for complex Linux incidents and root-cause analysis. The position manages the Linux support queue and on-call rotation, mentors junior and mid-level engineers, and collaborates with Security, Network, Database, Applications, and Quality teams to sustain secure, compliant, audit-ready systems in a 24x7x365 environment. The role leads Linux support and architecture, configuration management, automation, performance analysis, and lifecycle planning for an enterprise estate using RHEL and derivatives. It also owns engineering and support for Debian product OS releases, including image definition, kernel and device-tree configuration, package selection, hardening, and release qualification. The engineer supports Raspberry Pi image provisioning, unattended upgrades, patch delivery, and field issues in collaboration with Product Engineering, R&D, Manufacturing, and Field Service. The position develops Linux engineers and administrators and maintains SOPs, runbooks, and change/configuration records in accordance with company quality and regulatory requirements. The position is performed in a traditional office environment, with some local travel between office sites. Work may also require standing in data centers, data closets, and other secure environments and lifting equipment up to 50 pounds. Annual base salary is $116,600–$177,800; employees may qualify for a discretionary or non-discretionary bonus. MSD offers medical, dental, vision, prescription, retirement, leave, insurance, and other benefits.

Requirements

Bachelor's degree in Computer Science, Information Systems, or equivalent work experience.
At least 10 years of progressive Linux systems administration/engineering experience in production data center environments.
At least 3 years in a lead, principal, or equivalent senior technical capacity.
Experience building, hardening, and supporting an embedded or appliance Linux OS shipped as part of a product, including reproducible image creation, first-boot provisioning, and read-only or A/B root filesystem strategies.
Hands-on experience supporting databases on Linux, including Oracle, MySQL/MariaDB, PostgreSQL, or MongoDB.
Experience in production data center environments, including clustering, high availability, and disaster recovery for Linux workloads.
Experience operating in a regulated environment (GxP, ISO, or similar) with change control, documentation, and audit support.
Expert-level knowledge of enterprise Linux administration and internals, including RHEL, CentOS/Rocky/AlmaLinux, Debian (including Debian 13 "trixie"), and Ubuntu; kernel tuning, systemd, LVM, filesystems, networking, and performance diagnostics.
Knowledge of Linux server security hardening and tuning, including CIS benchmarks, SELinux, and vulnerability remediation.
Expert knowledge of physical and virtual server installation and configuration procedures.
Expert knowledge of patch and errata management at scale (Red Hat Satellite, Foreman, or equivalent), and backup and restore operations.
Expert knowledge of configuration management and automation at scale; Ansible is required, along with Terraform for infrastructure-as-code provisioning, Git-based version control, and CI/CD pipelines.
Hands-on Debian expertise, specifically Debian 13 ("trixie") on Raspberry Pi, including APT and dpkg package management, .deb packaging, apt pinning and preferences, unattended-upgrades, and local/private APT repository management.
Working knowledge of ARM64/aarch64 Linux and Raspberry Pi platform specifics, including boot chain and firmware, device tree overlays, GPIO/I2C/SPI and serial peripherals, SD/eMMC endurance, and thermal and power constraints.
Familiarity with product OS release qualification, remote upgrade and patch delivery to fielded devices, rollback strategy, and CVE tracking against a shipped software bill of materials.
Expertise in Shell and Python scripting for automation, tooling, and diagnostics.
Strong understanding of DNS, HTTP/S, LDAP, SSH, NFS, Kerberos, SMTP, and SNMP, including Active Directory integration of Linux hosts.
Understanding of basic network concepts and the OSI model.
Familiarity with web-related technologies, including web applications, web services, service-oriented architectures, and network/web-related protocols.
Ability to provide day-to-day guidance and mentoring in a consistently supportive and collaborative manner.
Advanced problem-solving and root-cause analysis skills, with composure while leading major incident response.
Strong verbal and written communication skills, including ability to brief senior leadership on risk, incidents, and roadmap.
Must be able to move/lift up to 50 pounds.

Nice To Haves

Experience with cloud hosting, SaaS, or PaaS technologies; AWS experience preferred.
Red Hat certification (RHCSA/RHCE) strongly preferred; cloud, security, or ITIL certifications are a plus.
VMware vSphere and container platforms (Docker, Kubernetes/OpenShift) experience preferred.
Perl scripting experience is a plus.

Responsibilities

Serve as technical lead and final escalation point for Linux support, owning triage, root-cause analysis, and permanent corrective action for complex incidents.
Own architecture, build standards, and lifecycle of the Linux estate (RHEL and derivatives), including capacity planning and hardware/OS refresh.
Define and enforce configuration management and infrastructure-as-code practices using Ansible, Terraform, Git, and CI/CD pipelines.
Provide engineering and support ownership for the Debian Linux OS distributed with company products, including image definition, kernel and device-tree configuration, package selection, hardening, and release qualification.
Partner with Product Engineering, R&D, Manufacturing, and Field Service on Raspberry Pi image provisioning at build, unattended upgrade and patch delivery mechanisms, and escalated field issues.
Lead performance analysis, fault isolation, and reporting of service exceptions across the Linux estate.
Partner with Network, Security, Windows, Database, and Application teams on data center builds, migrations, and cross-platform projects.
Mentor and develop Linux engineers and administrators, providing technical direction, training, and workload prioritization.
Author and maintain SOPs, runbooks, and change/configuration records in line with company quality and regulatory requirements.
Manage the Linux support queue and on-call rotation.
Perform other duties as assigned by the organization.