Senior Quality Engineer

OutSystems OutSystems · Enterprise · San Francisco Bay Area, United States

The Senior AI Quality Engineer will lead quality initiatives for OutSystems' AI-powered product ecosystem, focusing on defining and implementing test strategies for AI outputs and microservices. This role involves leveraging AI tools to enhance the SDLC, ensuring AI products are functional, reliable, ethical, and scalable, and establishing intelligent quality gates within CI/CD pipelines. The position requires expertise in AI toolchains, SDLC leadership, and modern testing frameworks, with a focus on delivering enterprise-grade AI solutions.

What you'd actually do

  1. Define and implement the end-to-end test strategy for major AI product areas.
  2. Provide technical leadership for complex automation challenges.
  3. Lead the team in designing coverage for AI-based features, including prompt injection testing, model hallucination checks, and performance benchmarking.
  4. Lead triage for all defects.
  5. Define and enforce "smart" quality gates within the CI/CD pipeline.

Skills

Required

  • Advanced experience using AI coding assistants (Claude Code, Cursor) and LLMs (Gemini, GPT-4) to accelerate the SDLC
  • Proven track record of managing product delivery in an Agile environment with a focus on CI/CD and automated quality gates
  • Proficiency in C#, Python or JavaScript and experience with modern testing frameworks (e.g., Playwright, PyTest)
  • Ability to drive initiatives from ideation to completion with minimal oversight
  • 5+ years in Quality Engineering or Software Development, with a significant focus on automated systems and AI-based applications

Nice to have

  • prompt injection testing
  • model hallucination checks
  • performance benchmarking

What the JD emphasized

  • strategic lead for quality within our AI-powered product ecosystem
  • defining how we validate non-deterministic AI outputs and complex microservices
  • defining and implementing the end-to-end test strategy for major AI product areas
  • technical leadership for complex automation challenges
  • designing coverage for AI-based features, including prompt injection testing, model hallucination checks, and performance benchmarking
  • analyze trends in model behavior and system performance
  • define and enforce "smart" quality gates within the CI/CD pipeline
  • hold responsibility for automated suites, ensuring they are reliable, efficient, and capable of catching regressions in both code and model output
  • lead quality initiatives for an entire product module or end-to-end user workflows, establishing the quality framework for key components of our core AI engine
  • solve systemic quality issues, balancing the mitigation of technical debt with the rapid pace of AI innovation
  • Advanced experience using AI coding assistants (Claude Code, Cursor) and LLMs (Gemini, GPT-4) to accelerate the SDLC
  • Proven track record of managing product delivery in an Agile environment with a focus on CI/CD and automated quality gates
  • Ability to drive initiatives from ideation to completion with minimal oversight, requiring only strategic alignment with leadership
  • build the blueprint for how AI-based products are delivered to millions of users
  • enterprise-grade framework
  • high-quality, reliable AI solutions
  • governed human-AI collaboration

Other signals

  • AI-powered product ecosystem
  • validate non-deterministic AI outputs
  • AI-augmented delivery
  • AI products are functional, reliable, ethical, and scalable
  • AI-First Test Strategy & Planning
  • AI-Augmented Automation
  • Advanced Test Design
  • Defect Management & Root Cause Analysis
  • Intelligent CI/CD Quality Gates
  • Strategic Reporting & Mentorship
  • AI Toolchain Expertise
  • AI-based applications
  • AI Development Platform
  • enterprise-grade framework
  • AI solutions
  • human-AI collaboration