Senior Production Engineer

Weights & Biases Weights & Biases · Data AI · Bellevue, WA +3 · Technology

Senior Production Engineer role focused on building and operating critical tooling for cloud reliability, performance, and operational excellence. This role involves hands-on ownership of systems, leading technical projects, and improving operational foundations through durable engineering investments. The engineer will ship production code and participate in incident response, focusing on distributed systems, cloud-native technologies, and observability stacks.

What you'd actually do

  1. Take hands-on ownership of critical systems and frameworks, driving their architecture, implementation, and long-term evolution.
  2. Lead end-to-end delivery of engineering projects that improve availability, scalability, operational automation, and failure recovery.
  3. Build and maintain observability, alerting, automated remediation, and resilience testing for the systems you support.
  4. Participate in incident response as a subject-matter expert; drive deep root-cause investigations and implement lasting fixes.
  5. Improve runbooks, sources of truth, deployment workflows, and operational tooling to harden production readiness.

Skills

Required

  • 7+ years of engineering experience building and operating distributed systems or cloud platforms.
  • Demonstrated ability to debug complex production issues end-to-end, across services, infrastructure layers, and automation.
  • Strong programming or scripting ability (Python, Go, or similar), with experience shipping and operating production services and tools.
  • Deep knowledge of cloud-native technologies and distributed system patterns, particularly Kubernetes.
  • Experience with modern observability stacks: metrics, tracing, structured logs, SLOs/SLIs, and incident lifecycle practices.
  • A track record of successfully delivering hands-on reliability improvements through engineering execution.

Nice to have

  • Experience building internal tooling, frameworks, or automation that supports high-availability cloud operations.
  • Familiarity with DR/BCP, service tiering, capacity planning, or chaos engineering.
  • Background operating or building large-scale AI or GPU-accelerated infrastructure.
  • Experience maintaining multi-year ownership of foundational production systems.

What the JD emphasized

  • building and operating distributed systems
  • cloud infrastructure at scale
  • reliability, performance, and operational excellence
  • observability, alerting, automated remediation, and resilience testing
  • debug complex production issues end-to-end