Senior AI Hardware Systems Engineer, Annapurna Labs, Trainium Machine Learning Fleet Operations

Amazon Amazon · Big Tech · Austin, TX · Software Development

This role focuses on the operational excellence and reliability of a fleet of ML accelerators and server products, specifically Amazon's Trainium chips. The engineer will be responsible for debugging hardware and software issues, developing automation, analyzing fleet data, and ensuring the health and performance of the ML hardware infrastructure. This is an engineering role focused on the operational aspects of serving ML hardware.

What you'd actually do

  1. Member of a team responsible for system remediation, operational excellence, and customer experience on bleeding edge ML products
  2. Utilize data to root cause hardware failures and identify live trends on the most complex systems in AWS
  3. Implement and improve system level testing across the product lifecycle
  4. Develop software which can be maintained, improved upon, documented, tested, and reused
  5. Dive deep on issues at the intersection of hardware and software

Skills

Required

  • 4+ years of programming with at least one modern language such as C++, C#, Java, Python, Golang, PowerShell, Ruby experience
  • 3+ years of non-internship professional software development experience, or Bachelor's degree or above in engineering or equivalent
  • 3+ years of designing or architecting (design patterns, reliability and scaling) of new and existing systems experience
  • Experience in computer architecture, or experience with general troubleshooting/debugging of hardware
  • Experience working with device technologies under development, familiarity with flashing firmware, basic device debugging and familiarity with reading/pulling device logs
  • BS degree in computer science, computer engineering, or related field, or 4+ years of technical work experience
  • 2+ years of server hardware troubleshooting and repair experience

Nice to have

  • 7+ years of leading design or architecture (design patterns, reliability and scaling) of new and existing systems experience
  • Experience with concepts such as system architecture, optimization, system dynamics, system analysis, statistical analysis, reliability analysis, and decision making
  • Experience that includes strong analytical skills, attention to detail, and effective communication abilities, or experience troubleshooting and debugging technical systems and experience with automation and any version control tools
  • Knowledge of operating systems, hardware, storage, network, security, database administration and cloud infrastructure
  • Master's degree or above in electrical engineering, computer engineering, or equivalent
  • Experience with SOC bring-up and post-silicon validation

What the JD emphasized

  • debug emergent problems in GPU and server hardware
  • running large scale experiments on a fleet of complex hardware
  • develop data infrastructure and analyzing trends
  • build automation software to scale operations
  • end to end ownership of some of the most advanced server hardware in the world
  • drive technical debug efforts
  • write truly massive scale autonomous software to monitor, optimize, and remediate machine learning hardware
  • maximize its health, sellability, and customer experience
  • partnering with hardware and software engineering teams to debug, investigate, and translate findings into permanent fixes
  • own the end-to-end testing story
  • manage tradeoffs between coverage and velocity
  • direct new automations, tooling, and data infrastructure to scale your operations
  • manage software deployments, debug issues with them, and run status meetings to align all platform stakeholders on how the product is performing
  • maintain an exceptionally high quality bar for our fleet of advanced machine learning accelerators and server products
  • perfect the customer experience by developing scalable software for rapid incident response times and data visualization
  • diving deep into hardware issues as they arise

Other signals

  • ML hardware fleet operations
  • debug emergent problems in GPU and server hardware
  • running large scale experiments on a fleet of complex hardware
  • developing data infrastructure and analyzing trends
  • building automation software to scale operations