Senior Production Engineer

Weights & Biases Weights & Biases · Data AI · Bellevue, WA +3 · Technology

Senior Production Engineer role focused on ensuring cloud reliability, performance, and operational excellence through hands-on ownership of critical tooling. Responsibilities include designing, implementing, and operating systems to improve delivery velocity, service availability, and operational safety, leading technical projects, and strengthening operational foundations. Requires experience in building and operating distributed systems, debugging complex production issues, and cloud-native technologies like Kubernetes. Experience with AI/GPU infrastructure is a plus.

What you'd actually do

  1. Take hands-on ownership of critical systems and frameworks, driving their architecture, implementation, and long-term evolution.
  2. Lead end-to-end delivery of engineering projects that improve availability, scalability, operational automation, and failure recovery.
  3. Build and maintain observability, alerting, automated remediation, and resilience testing for the systems you support.
  4. Participate in incident response as a subject-matter expert; drive deep root-cause investigations and implement lasting fixes.
  5. Improve runbooks, sources of truth, deployment workflows, and operational tooling to harden production readiness.

Skills

Required

  • 7+ years of engineering experience building and operating distributed systems or cloud platforms.
  • Demonstrated ability to debug complex production issues end-to-end, across services, infrastructure layers, and automation.
  • Strong programming or scripting ability (Python, Go, or similar), with experience shipping and operating production services and tools.
  • Deep knowledge of cloud-native technologies and distributed system patterns, particularly Kubernetes.
  • Experience with modern observability stacks: metrics, tracing, structured logs, SLOs/SLIs, and incident lifecycle practices.
  • A track record of successfully delivering hands-on reliability improvements through engineering execution.

Nice to have

  • Experience building internal tooling, frameworks, or automation that supports high-availability cloud operations.
  • Familiarity with DR/BCP, service tiering, capacity planning, or chaos engineering.
  • Background operating or building large-scale AI or GPU-accelerated infrastructure.
  • Experience maintaining multi-year ownership of foundational production systems.

What the JD emphasized

  • hands-on ownership
  • critical systems
  • end-to-end technical projects
  • long-lived systems
  • building
  • debugging
  • operating production systems
  • end-to-end
  • shipping and operating production services and tools
  • foundational production systems