Software Engineer, Reliability

OpenAI OpenAI · AI Frontier · San Francisco, CA · Applied AI

Software Engineer focused on reliability for OpenAI's AI systems, ensuring scalability, performance, and stability of infrastructure. This role involves building tools for load/chaos/synthetic testing, automation, resource lifecycle management, and implementing fault-tolerant designs. The engineer will also define SLOs/SLIs, participate in on-call rotations, and collaborate with research and product teams to bring AI technology to users safely and reliably.

What you'd actually do

  1. Design and implement solutions to ensure the scalability of our infrastructure to meet rapidly increasing demands.
  2. Build and maintain the load, chaos and synthetic testing software leveraged by development teams to make the systems they design and operate more reliable.
  3. Build and maintain automation tools to streamline repetitive tasks and improve system reliability.
  4. Build and maintain the platform for CPU/storage, GPU, and network lifecycle management to drive efficiency, accountability and support dynamic optimization of our resources.
  5. Implement fault-tolerant and resilient design patterns to minimize service disruptions.

Skills

Required

  • Proven experience as an SWE focused on reliability or a similar role in a fast-paced, rapidly scaling company.
  • Strong proficiency in cloud infrastructure.
  • Proficiency in programming languages.
  • Experience with containerization technologies and container orchestration platforms like Kubernetes.
  • Knowledge of IaC tools such as Terraform or CloudFormation.
  • Excellent problem-solving and troubleshooting skills.
  • Strong communication and collaboration skills.
  • Experience with observability tools such as DataDog, Prometheus, Grafana and Splunk.
  • Experience with microservices architecture and service mesh technologies.
  • Knowledge of security best practices in cloud environments.

Nice to have

  • Have a track record of accelerating engineering reliability by empowering your fellow engineers with excellent tooling and systems.
  • Have a humble attitude, an eagerness to help your colleagues, and a desire to do whatever it takes to make the team succeed.
  • Own problems end-to-end, and are willing to pick up whatever knowledge you're missing to get the job done.
  • Enjoy seeking out and addressing bottlenecks and areas for performance improvement in our systems.
  • Utilize Infrastructure as Code (IaC) principles to automate infrastructure provisioning and configuration management.
  • Are experienced in collaborating with cross-functional teams to ensure that reliability and scalability are considered in the design and development of new features and services.

What the JD emphasized

  • ensure the reliability, scalability, and performance of our systems
  • reliability, scalability, and performance