What you'd actually do

Apply Docusign's AI testing framework to evaluate LLM-powered features and agentic AI workflows (e.g., Quality Agent, XDR Agents, Leads Agent)

Design and execute prompt test suites to validate LLM output accuracy, consistency, tone, and alignment with expected outcomes across diverse input scenarios

Conduct hallucination detection and post-processing fact-checking for LLM-generated content; validate that AI outputs are grounded in retrieved context

Validate training/validation datasets for missing values, data leakage, class imbalance, duplicates, and biased sampling

Execute model quality testing using metrics such as accuracy, precision/recall, F1, ROC-AUC, and regression error (MAE/RMSE) as applicable to the model type

Skills

Required

design, develop, and maintain scalable test automation frameworks
AI testing methodologies
LLM evaluation
prompt testing
hallucination detection
AI-assisted STLC
evaluator frameworks (e.g., Arize AX)
agentic AI workflows
prompt test suites
model quality testing
training/validation datasets
RAG-based AI features
performance and load testing strategies
CI/CD integration
data validation automation

Nice to have

Salesforce
Oracle ERP
MuleSoft integrations
UiPath workflow automation
CrewAI
Azure OpenAI
Azure AI Search
JMeter
k6
GitLab CI

Company Overview Docusign brings agreements to life. Over 1.5 million customers and more than a billion people in over 180 countries use Docusign solutions to accelerate the process of doing business and simplify people’s lives. With intelligent agreement management, Docusign unleashes business-critical data that is trapped inside of documents. Until now, these were disconnected from business systems of record, costing businesses time, money, and opportunity. Using Docusign’s Intelligent Agreement Management platform, companies can create, commit, and manage agreements with solutions created by the #1 company in e-signature and contract lifecycle management (CLM). What you'll do We are seeking a skilled and independent Software Development Engineer in Test (SDET) at the P3 (Career) level to join our Quality Engineering team in Bengaluru. In this role, you will design, develop, and maintain scalable test automation frameworks across enterprise SaaS platforms — including Salesforce, Oracle ERP, MuleSoft integrations, and UiPath workflow automation — ensuring the highest levels of product quality across the end-to-end digital and direct customer journey. A critical differentiator of this role is your ability to independently apply AI testing methodologies — including LLM evaluation, prompt testing, hallucination detection, and AI-assisted STLC practices — to ensure quality for both traditional and AI-powered product features. You will leverage Docusign's internal AI quality tooling (including the Quality Agent powered by CrewAI) and evaluator frameworks (such as Arize AX) to accelerate test delivery and improve coverage consistency. This is an individual contributor role reporting to the Senior Manager, SDET. Responsibility Design, develop, and maintain scalable and reusable test automation frameworks for enterprise applications across Salesforce Sales, Service & Experience Clouds, Oracle ERP, and MuleSoft integrations Independently own test coverage across multiple product components including UI, API, backend, data validation, integration, and regression testing Develop and execute comprehensive test plans, test cases, and test scripts for end-to-end business flows aligned to acceptance criteria Drive shift-left quality practices by partnering with developers and platform architects early in the SDLC to identify testability gaps and integrate quality into design Own test coverage metrics, dashboarding, and quality reporting to leadership Perform exploratory, functional, SIT, regression, and UAT testing for critical releases; support production smoke testing and hypercare post go-live Identify requirement gaps, provide robust edge cases and error-handling scenarios beyond documented functionality, and contribute to continuous quality improvemen Apply Docusign's AI testing framework to evaluate LLM-powered features and agentic AI workflows (e.g., Quality Agent, XDR Agents, Leads Agent) Design and execute prompt test suites to validate LLM output accuracy, consistency, tone, and alignment with expected outcomes across diverse input scenarios Conduct hallucination detection and post-processing fact-checking for LLM-generated content; validate that AI outputs are grounded in retrieved context Validate training/validation datasets for missing values, data leakage, class imbalance, duplicates, and biased sampling Execute model quality testing using metrics such as accuracy, precision/recall, F1, ROC-AUC, and regression error (MAE/RMSE) as applicable to the model type Verify model behavior under noisy, incomplete, adversarial, or out-of-distribution inputs Validate prompt-injection resistance, jailbreak detection, harmful/toxic content filtering, and secure handling of sensitive data and PII Implement AI evaluator frameworks using tools such as Arize AX for offline evaluation (datasets, experiments) and online testing (tracing, drift monitoring, confidence scoring) Track hallucination rates, latency variability, token cost, and model confidence thresholds in collaboration with MLOps teams Apply human-in-the-loop (HITL) evaluation practices to systematically assess AI output quality — helpfulness, correctness, compliance — where automated metrics are insufficient Leverage Docusign's internal Quality Agent (CrewAI-powered, Azure OpenAI-backed) to accelerate test case generation from Jira user stories, reducing manual effort and improving STLC throughput Test retrieval relevance, semantic search accuracy (Azure AI Search), context window utilization, and end-to-end response quality for RAG-based AI features Build and maintain robust, reusable automation suites for functional, regression, integration, performance, and data validation testing Automate end-to-end business flows using UiPath or equivalent low-code/RPA automation platforms; build solutions that reduce manual effort for repetitive processes Integrate automated tests into CI/CD pipelines (GitLab CI or equivalent) to support rapid, reliable releases and continuous quality gates Develop and maintain data validation automation scripts; perform source-to-target validation, data quality checks (accuracy, completeness, consistency, timeliness), and business rule validations on platforms such as Snowflake Implement performance and load testing strategies (e.g., JMeter, k6) for high-volume data operations and enterprise application integrations Conduct code reviews with a focus on test quality, reliability, and maintainability; contribute to continuous improvement of the automation system Partner closely with product managers, business analysts, developers, and platform architects to align testing strategy to business outcomes and delivery goals Participate in requirements workshops, sprint ceremonies, change management processes, and go-live support activities Proactively communicate quality risks, test coverage gaps, defect trends, and improvement recommendations to team and leadership Mentor junior SDET and QA team members in automation best practices, AI testing tools, and low-code platform usage Contribute to standardizing quality practices and reporting on overall application quality metrics and KPIs across programs (Digital, Direct, Customer Success, GTM) Job Designation Hybrid: Employee divides their time between in-office and remote work. Access to an office location is required. (Frequency: Minimum 2 days per week; may vary by team but will be weekly in-office expectation) Positions at Docusign are assigned a job designation of either In Office, Hybrid or Remote and are specific to the role/job. Preferred job designations are not guaranteed when changing positions within Docusign. Docusign reserves the right to change a position's job designation depending on business needs and as permitted by local law. What you bring Basic Bachelor's degree in Computer Science, Engineering, or a related field (Master's preferred) 5+ years of experience in QA/SDET roles (or 3+ years with a Master's degree), with at least 3 years focused on test automation for enterprise SaaS applications Strong proficiency in Python for building automation frameworks and AI evaluation scripts Hands-on experience with UI automation tools (UiPath, Selenium, Playwright) and API testing tools (Postman, RestAssured) Hands-on experience with Salesforce platform testing, including Apex testing across Sales, Service, and Experience Clouds Experience integrating automation into CI/CD pipelines (GitLab CI or equivalent) with Git-based version control Strong SQL proficiency for data validation, source-to-target testing, and business rule verification Working knowledge of AI/LLM testing fundamentals: prompt testing, hallucination detection, model quality metrics (F1, accuracy, precision/recall), and AI safety testing concepts Strong understanding of QA best practices, testing methodologies, and Agile/Scrum frameworks Proven experience with Jira and Zephyr Scale for test management and traceability Preferred Hands-on experience with Arize AX or similar LLM observability/evaluation platforms (offline datasets, experiments, online tracing, and drift monitoring) Experience with CrewAI, LangChain, or similar agentic AI orchestration frameworks and approaches to testing multi-agent workflows Familiarity with RAG pipeline testing: retrieval relevance evaluation, vector database validation (Azure AI Search), and semantic search quality assessment Experience leveraging Docusign's internal Quality Agent for AI-assisted test case generation and STLC automation Knowledge of MLOps principles and testing methodologies for validating and monitoring ML models in production (e.g., data drift detection, model bias testing, A/B evaluation) Experience with performance testing tools such as JMeter, k6, Gatling, or Blazemeter Active Salesforce certifications (e.g., Platform Developer I, Administrator)Strong defect triage, debugging, and root cause analysis skills Life at Docusign Working here Docusign is committed to building trust and making the world more agreeable for our employees, customers and the communities in which we live and work. You can count on us to listen, be honest, and try our best to do what’s right, every day. At Docusign, everything is equal. We each have a responsibility to ensure every team member has an equal opportunity to succeed, to be heard, to exchange ideas openly, to build lasting relationships, and to do the work of their life. Best of all, you will be able to feel deep pride in the work you do, because your contribution helps us make the world better than we found it. And for that, you’ll be loved by us, our customers, and the world in which we live. Accommodation Docusign is committed to providing reasonable accommodations for qualified individuals with disabilities in our job application procedures. If you need such an accommodation, or a religious accommodation, during the application process, please contact us at accommodations@docusign.com. If you experience any issues, concerns, or technical difficulties during the application process please get in touch with our Talent organization at taops@docusign.com for assistance. Applicant and Candidate Privacy Notice #LI-Hybrid #LI-SA4

Sdet

What you'd actually do

Skills

Required

Nice to have

What the JD emphasized

Other signals