Senior Server Operations & Maintenance Engineer

ByteDance ByteDance · Big Tech · San Jose, CA · Infrastructure

Seeking a Senior Server Operations & Maintenance Engineer to manage the full lifecycle of server operations, including new product introduction, quality management, and platform system planning. Responsibilities include formulating monitoring plans, analyzing complex hardware issues, building a knowledge base, and implementing AIOps for anomaly detection.

What you'd actually do

  1. Responsible for the operation management of server new product introduction and the construction of an adaptation system. This includes formulating server operation and maintenance standard baselines, server hardware monitoring and formulation, planning, designing, iteratively optimizing the maintainability, ease of maintenance, and stability solutions for new products, identifying and evaluating operation risks during new product introduction, and constructing and adapting new product operation and maintenance solutions.
  2. Deeply involved in formulating hardware monitoring plans and verifying and launching monitoring rules for new online models, new hardware, and new scenarios. Precisely define and continuously improve various hardware failure judgment criteria, enhance the accuracy and recall rate of hardware monitoring, and ensure the stable operation of servers.
  3. For complex hardware troubleshooting of online servers, conduct in - depth analysis based on server - level black - box logs (such as BMC logs, IPMI/SMBIOS data, operating system kernel logs, hardware event logs, etc.). Collaborate with internal and external software and hardware R & D experts to accurately locate the root causes of difficult problems. Continuously improve the clarity rate of hardware failure monitoring and diagnosis and the consistency rate between repair reports and actual repairs, effectively reducing the failure return rate.
  4. Build a hardware failure knowledge base, accumulate typical cases and troubleshooting methodologies, promote the upgrade of monitoring automation and intelligence (such as the implementation of AIOps anomaly detection), participate in the construction of the hardware monitoring system and the standardization of hardware operation and maintenance processes, and write and maintain technical documents and operation manuals.
  5. Online quality operation, Establish unified quality standards, clarify the basis for batch quality determination, define the collaborative interface for batch quality issue repair, accumulate historical batch issue experience, implement risk management in advance, and control risks from the source.

Skills

Required

  • Bachelor’s degree or higher in Computer Engineering, Computer Science or related majors
  • Rich experience in operation and maintenance of large-scale servers in the Internet industry
  • Familiar with Internet operation and maintenance process system
  • Familiar with server operation and maintenance throughout its life cycle
  • Familiar with server hardware products development and delivery processes
  • Familiar with server operation and maintenance scenarios
  • Proficiently master risk identification for server maintainability, ease of maintenance, and stability
  • Familiar with server hardware architecture (x86/ARM, PCIe/NVMe, CPU, Disk, Memory, GPU/NPU, NIC, AOC)
  • Familiar with BMC/IPMI protocol, Redfish, Sensor, SEL log, BMC OneKeyLog, GPU stress test diagnosis
  • Proficient in Linux system
  • Skilled in server hardware configuration and troubleshooting under Linux
  • Proficient in Shell scripting, Python, PHP, Perl, or Lua

Nice to have

  • Full life cycle management process of servers
  • 3+ years of experience in hardware research and development and testing
  • Server quality management work
  • 3+ years of experience in online quality management
  • Server hardware monitoring work
  • 3+ years of experience in server hardware index collection, reporting, and monitoring strategy threshold operation

What the JD emphasized

  • rich experience in the operation and maintenance of large-scale servers in the Internet industry
  • familiar with the Internet operation and maintenance process system
  • familiar with the operation management of the server throughout its life cycle
  • Familiar with server hardware architecture
  • proficiently master the ability to identify risks related to server maintainability, ease of maintenance, and stability