Senior Platform and Engops Engineer - Cluster Operations

NVIDIA NVIDIA · Semiconductors · Santa Clara, CA

NVIDIA is seeking a Senior Platform and EngOps Engineer to develop and maintain software for GPU clusters, focusing on automation, deployment, provisioning, and maintenance of large-scale GPU infrastructure interconnected via NVLink and InfiniBand. The role involves managing cluster failures, software updates, and collaborating with engineering teams.

What you'd actually do

  1. Develop automated tools to efficiently deploy, provision, and maintain extensive GPU clusters interconnected via NVLink and InfiniBand
  2. Implement modern DevOps tools to automate software updates, perform maintenance tasks, and monitor cluster availability, ensuring seamless operations.
  3. Take ownership of daily cluster failures and issues, troubleshooting them promptly to maintain optimal cluster availability and performance.
  4. Manage the rollout and rollback of cluster software and firmware updates, ensuring smooth transitions and minimal disruptions.
  5. Collaborate effectively with dynamic Engineering and Product Teams across multiple time zones to align cluster operations with evolving project requirements.

Skills

Required

  • BS or MS in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.
  • 8+ years of hands-on experience in deploying and administrating clusters, servers, switches, and related infrastructure.
  • Automation expert with hands on skills in Ansible, Python and Shell Scripting.
  • Deep understanding of operating systems, computer networks, and high-performance applications.
  • Proven ability to work effectively with developers and test engineers across different teams and time zones.
  • Proficient with Linux fundamentals.

Nice to have

  • Familiarity with resource scheduling managers, preferably Slurm.
  • Direct experience with industry standard alerting tools and emergency response practices.
  • Hands-on experience with GPU-focused hardware and software, such as DGX systems and Compute Clusters.
  • Proficiency in crafting and implementing a robust metrics collection and alerting infrastructure.
  • Proficiency in designing large scale networking technologies and the associated challenges.

What the JD emphasized

  • 8+ years of hands-on experience in deploying and administrating clusters, servers, switches, and related infrastructure.
  • Proven ability to work effectively with developers and test engineers across different teams and time zones.