Software Engineering Smts

Salesforce Salesforce · Enterprise · San Francisco, CA

Software Quality Senior Member of Technical Staff (SMTS) at Salesforce, focusing on AI-first testing frameworks and automation for AI-driven Sales Agent and Forecasting capabilities. The role involves validating LLM integrations, prompt accuracy, and human-in-the-loop evaluation processes for subjective quality dimensions of AI features. Requires strong Java skills and an AI/LLM validation mindset, with experience in CI/CD environments.

What you'd actually do

  1. Design, build, and maintain scalable, AI-first testing frameworks and automation suites to validate complex RAO workflows and agentic capabilities.
  2. Translate high-level quality strategies into actionable automation plans, focusing heavily on validating LLM integrations, prompt accuracy, and deterministic system behaviors.
  3. Dig deep into the application code to understand system architecture, identify quality gaps, perform white-box testing, and actively contribute to fixing code and test flakiness.
  4. Implement and support evaluation processes for subjective quality dimensions of our AI features, including response relevance, user safety, and contextual accuracy.
  5. Partner closely with developers, product managers, and engineers within the RAO ecosystem to ensure quality is integrated early and continuously throughout the agile delivery lifecycle.

Skills

Required

  • Java
  • data structures
  • algorithms
  • CI/CD
  • automation

Nice to have

  • AI/LLM validation
  • GPT-like models
  • AI evaluation frameworks
  • white-box testing
  • public cloud infrastructure
  • Salesforce platform
  • autonomous agents
  • conversational UIs
  • generative AI workflows

What the JD emphasized

  • AI-first testing frameworks
  • validating LLM integrations
  • human-in-the-loop
  • evaluation processes
  • agentic capabilities

Other signals

  • AI-first testing frameworks
  • validating LLM integrations
  • human-in-the-loop evaluation
  • AI-driven Sales Agent and Forecasting solutions