Cloud Production Platform Engineer

Meta•DeKalb, IL

4d•$173,000 - $245,000

About The Position

Meta is seeking an experienced hardware engineering professional with technical leadership experience and skills in Server Hardware, high performance Compute, Storage, Accelerators (GPU), and/or Networking, ideally in a data center environment. Our Production Platform Engineers are responsible for the performance of the compute, storage, and accelerator (GPU) platforms that run Meta's production workloads, both in our own data centers and on capacity Meta consumes from cloud providers. They are responsible for the health of the server fleet, from NPI production operational test through end of life. Responsibilities include identifying systemic hardware, firmware, and tooling issues; engaging in hands-on problem solving; and collaborating effectively with engineering, tooling, and onsite provisioning and break fix teams to improve performance of the fleet. This position carries the cloud platform scope for the team. Meta's cloud footprint is growing toward the scale of the fleet we run ourselves, across several cloud providers, and product groups expect the same reliability and availability from it. Because Meta does not design this hardware and does not have physical access to it, the engineer in this role works through each cloud provider's own repair and escalation process, reaches diagnostic conclusions with partial telemetry, and builds the technical evidence that supports cloud provider accountability. This remains a platform engineering role across both environments. It is not a cloud only role. This position requires a high degree of understanding of our platforms, leveraging technical expertise and using data to identify trends across the fleet and proactively identify and mitigate performance and reliability issues. Experience organizing and prioritizing work across multiple teams in a large-scale, distributed environment and experience partnering with cross-functional teams to drive outcomes in a large-scale, distributed environment.

Requirements

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
BS or BA in electrical, mechanical, or computer engineering, or equivalent hardware-engineering-related experience
10+ years of experience with server hardware systems in an internet operations environment, or on a development team in design or test validation
Experience triaging and debugging hardware platforms
Familiarity or experience in NPI server platform development
Familiarity with Linux or Unix Operating systems
Experience training and mentoring technicians and/or engineers
Demonstrated technical skills and experience creating documentation for users of all levels
Communication experience and interpersonal skills, with the experience communicating and partnering effectively within all levels of the organization
Experience with building relationships with customers and vendors
Expert at leveraging large volumes of data to improve decision making processes in the hardware functional domain (understanding the metrics and statistics)

Nice To Haves

Experience in data center system and process automation
Experience in large scale data center hardware deployments and building scaling infrastructure
BS or BA in electrical, mechanical, or computer engineering
Experience supporting server hardware in a public cloud, hosted, or colocation environment, including working through a cloud provider's repair and escalation process
Experience diagnosing hardware issues with limited physical access and partial telemetry
Experience engaging technically with hardware vendors or service providers, including evidence based escalation and service level discussions
Bash, PHP, Python, or Perl scripting experience
Experience in integration testing and operation of compute, storage, and accelerator (GPU) platforms, bringing operational perspective to each stage of H/W development life cycle through end of life in production

Responsibilities

Serve as subject matter expert for compute, storage, and accelerator (GPU) platforms in Meta's production data centers and on capacity Meta consumes from cloud providers, anticipating and diagnosing issues for the operations teams, and escalating as appropriate to hardware design and NPI teams
Act as the deep escalation point for cloud hardware issues, taking handoffs from the cloud triage teams and driving hardware and software investigation through to resolution
Identify systemic hardware issues within a single cloud provider and across cloud providers, separating cloud provider specific behavior from platform behavior already characterized in Meta's own fleet
Represent Meta's reliability, diagnosability, and serviceability expectations to cloud providers, compiling the technical evidence those conversations depend on
Carry findings from the cloud environment back into Meta's own platform work, and apply what is already known about Meta's platforms to accelerate diagnosis in the cloud
Provide end to end lifecycle support (operational test through end of life decommissioning) of hardware platforms and new technologies in the data centers
Drive complex technical investigations globally and spanning multiple disciplines such as Hardware, Software/Firmware, Networking and Power & Cooling
Analyze health of the global fleet of servers to proactively identify systemic issues, and take appropriate action to quickly mitigate impact
Work closely with HW Engineering, Release to Production, and Systems Integration teams to take new server and storage platforms into mass production in the data centers
Identify risks early in new hardware programs and contribute the test methods needed to characterize the product and assure end customer needs are met
Act as product owner for specific server and/or storage platforms. Perform and drive RCA and CA, in collaboration with hardware and software engineering teams
Issue timely alerts and fixes to operations teams, and assure a robust feedback pipeline to engineering teams
Build strong relationships and collaboration with engineering and cross functional teams across the company. Actively solicit feedback from teams, and use that feedback to improve operational effectiveness as infrastructure scales
Provide serviceability feedback on production hardware resulting in more efficient repair activities