Software Engineering Smts

Salesforce Salesforce · Enterprise · San Francisco, CA +1

Salesforce is seeking a Senior Member of Technical Staff (SMTS) in Software Engineering to focus on AI-first testing and validation for their Product Led Growth organization. The role involves building and maintaining scalable AI-first testing frameworks, validating LLM integrations, prompt accuracy, and system behaviors. It also includes implementing human-in-the-loop evaluation processes for AI features and collaborating with cross-functional teams to ensure high quality for enterprise AI-driven solutions.

What you'd actually do

  1. Design, build, and maintain scalable, AI-first testing frameworks and automation suites to validate complex PLG workflows and agentic capabilities.
  2. Translate high-level quality strategies into actionable automation plans, focusing heavily on validating LLM integrations, prompt accuracy, and deterministic system behaviors.
  3. Dig deep into the application code to understand system architecture, identify quality gaps, perform white-box testing, and actively contribute to fixing code and test flakiness.
  4. Implement and support evaluation processes for subjective quality dimensions of our AI features, including response relevance, user safety, and contextual accuracy.
  5. Partner closely with developers, product managers, and engineers within the PLG ecosystem to ensure quality is integrated early and continuously throughout the agile delivery lifecycle.

Skills

Required

  • Strong Java Skills
  • AI/LLM Validation Mindset
  • Agile Delivery at Pace
  • Collaboration & Communication
  • Problem-Solving & Analytical Skills

Nice to have

  • Experience testing autonomous agents
  • Experience testing conversational UIs
  • Experience testing generative AI workflows
  • Experience with public cloud infrastructure (AWS/Azure/GCP)
  • Foundational knowledge of the Salesforce platform
  • Strong experience with white-box testing
  • Fixing code

What the JD emphasized

  • AI-first testing frameworks
  • validating LLM integrations
  • human-in-the-loop evaluation
  • AI-driven solutions

Other signals

  • AI-first testing frameworks
  • validating LLM integrations
  • human-in-the-loop evaluation
  • AI-driven solutions quality standards