Senior Production Engineer

Weights & Biases Weights & Biases · Data AI · Singapore · Technology

Senior Production Engineer role focused on building and operating critical tooling for cloud reliability, performance, and operational excellence. Responsibilities include system design, implementation, deployment, maintenance, incident response, and improving operational foundations. Requires strong experience in distributed systems, cloud-native technologies (Kubernetes), and observability stacks. Experience with AI/GPU infrastructure is a plus.

What you'd actually do

  1. Take hands-on ownership of critical systems and frameworks, driving their architecture, implementation, and long-term evolution.
  2. Lead end-to-end delivery of engineering projects that improve availability, scalability, operational automation, and failure recovery.
  3. Build and maintain observability, alerting, automated remediation, and resilience testing for the systems you support.
  4. Participate in incident response as a subject-matter expert; drive deep root-cause investigations and implement lasting fixes.
  5. Improve runbooks, sources of truth, deployment workflows, and operational tooling to harden production readiness.

Skills

Required

  • building and operating distributed systems
  • cloud platforms
  • debugging complex production issues
  • Python
  • Go
  • shipping and operating production services and tools
  • cloud-native technologies
  • distributed system patterns
  • Kubernetes
  • observability stacks (metrics, tracing, structured logs, SLOs/SLIs)
  • incident lifecycle practices
  • delivering reliability improvements through engineering execution

Nice to have

  • internal tooling, frameworks, or automation for high-availability cloud operations
  • DR/BCP
  • service tiering
  • capacity planning
  • chaos engineering
  • operating or building large-scale AI or GPU-accelerated infrastructure
  • multi-year ownership of foundational production systems

What the JD emphasized

  • hands-on ownership
  • end-to-end technical projects
  • long-lived systems
  • durable engineering investments
  • building, debugging, and operating production systems
  • primary impact comes from the reliability, quality, and maturity of the systems you deliver and maintain over time
  • hands-on ownership of critical systems and frameworks
  • Lead end-to-end delivery of engineering projects
  • Build and maintain observability, alerting, automated remediation, and resilience testing
  • Participate in incident response as a subject-matter expert
  • drive deep root-cause investigations and implement lasting fixes
  • Improve runbooks, sources of truth, deployment workflows, and operational tooling
  • Eliminate single points of failure and reduce operational toil through automation, refactors, and system redesigns
  • Ship production code regularly
  • participate in on-call rotations
  • Maintain and mature long-term projects and frameworks owned by the team
  • ensuring they remain reliable, well-instrumented, and easy to operate
  • Collaborate with platform teams to ensure new features and services integrate cleanly with our reliability best-practices and tooling
  • 7+ years of engineering experience building and operating distributed systems or cloud platforms.
  • Demonstrated ability to debug complex production issues end-to-end, across services, infrastructure layers, and automation.
  • Strong programming or scripting ability (Python, Go, or similar), with experience shipping and operating production services and tools.
  • Deep knowledge of cloud-native technologies and distributed system patterns, particularly Kubernetes.
  • Experience with modern observability stacks: metrics, tracing, structured logs, SLOs/SLIs, and incident lifecycle practices.
  • A track record of successfully delivering hands-on reliability improvements through engineering execution.
  • Experience building internal tooling, frameworks, or automation that supports high-availability cloud operations.
  • Background operating or building large-scale AI or GPU-accelerated infrastructure.
  • Experience maintaining multi-year ownership of foundational production systems.