Senior Production Engineer - Dgx Cloud

NVIDIA NVIDIA · Semiconductors · CA +5 · Remote

NVIDIA is seeking experienced Senior Production Engineers to scale its AI Infrastructure, focusing on production systems for large GPU clusters used in AI workloads. The role involves implementing monitoring, health management, and ensuring reliability and scalability of GPU assets, working with various data streams and cross-functional teams.

What you'd actually do

  1. You will be part of an DGX Cloud team responsible for production systems that enable large scalable GPU clusters to be used for a variety of AI workloads. This includes working on custom software related to GPU asset provisioning, configuration, and lifecycle management across cloud providers.
  2. Implementing monitoring and health management capabilities that enable industry leading reliability, availability, and scalability of GPU assets. You will be harnessing multiple data streams, ranging from GPU hardware diagnostics to cluster and network telemetry.
  3. Working with teams across NVIDIA to ensure production AI clusters run reliability and consistently with maximum performance. Evaluating system failures and improving services based on a well-defined incident management process.

Skills

Required

  • site reliability principles and techniques
  • reliability assessments
  • incident management processes
  • production system observability
  • monitoring and alerting
  • automated deployments
  • toil elimination
  • software engineering discipline
  • systems programming language (Go, Python)
  • data structures and algorithms
  • large-scale production systems

Nice to have

  • managing and automating large-scale distributed systems independent of cloud providers
  • cluster management systems (Kubernetes, Slurm, Bright Cluster Manager)

What the JD emphasized

  • significant experience with site reliability principles and techniques
  • significant contributions to our codebase
  • 8+ years in similar role and experience on large-scale production systems
  • Proven operational excellence in maintaining reliable and performant AI infrastructure

Other signals

  • production systems
  • AI infrastructure
  • GPU clusters
  • reliability
  • observability
  • monitoring
  • automated deployments