Senior Software Engineer - Experience Agents

Qualtrics Qualtrics · Seattle · Seattle, WA · Core AI

Senior Software Engineer focused on building observability, testing, and confidence systems for AI agents within Qualtrics' Agent Studio. The role involves developing real-time visibility into agent performance, implementing test coverage frameworks, and partnering with science teams on LLM evaluation metrics and debugging tools. This position is crucial for enabling customers to confidently deploy AI agents and ensuring the platform scales reliably.

What you'd actually do

  1. Build and evolve monitoring systems that give customers real-time visibility into agent performance, enabling them to measure success and identify improvement areas
  2. Design and implement test coverage frameworks that give customers confidence their agents are production-ready before deployment
  3. Partner with the science team to develop LLM evaluation metrics and debugging tools that help customers optimize agent behavior
  4. Collaborate with forward deployment engineers and customers to identify monitoring gaps and translate feedback into product improvements
  5. Contribute to the platform's transition from incubation to general availability by ensuring monitoring and testing scale reliably across growing customer base

Skills

Required

  • 5+ years of software development experience with demonstrated ownership of features or systems from conception through production
  • Experience building or working with monitoring, observability, or testing systems at scale
  • Proven ability to work across full-stack domains (backend and frontend) in a fast-moving environment
  • Track record of learning new technical domains quickly and shipping quality work
  • Experience working with AI/ML tools or systems, or demonstrated ability to pick up new technical areas rapidly

Nice to have

  • Writes effective prompts and debugs LLM graphs using tools like LangSmith
  • Uses AI tools like Claude and Cursor fluently to accelerate development and problem-solving
  • Communicates clearly with both technical and non-technical stakeholders, translating monitoring insights into actionable customer guidance
  • Demonstrates customer obsession by prioritizing metrics and insights that help customers succeed

What the JD emphasized

  • build the observability and confidence systems
  • measure success and iterate rapidly
  • writes effective prompts and debugs LLM graphs
  • monitoring, observability, or testing systems at scale
  • working with AI/ML tools or systems

Other signals

  • building observability and confidence systems for AI agents
  • enabling customers to confidently deploy AI agents
  • measure success and iterate rapidly
  • develop LLM evaluation metrics and debugging tools
  • optimize agent behavior
  • ensure monitoring and testing scale reliably