Testing Manager – Analytics & AI Evaluation Center of Excellence Lead

JPMorgan Chase JPMorgan Chase · Banking · Bengaluru, Karnataka, India · Corporate Sector

Lead the AI Evaluation Center of Excellence, focusing on User Acceptance Testing (UAT) and business validation for HR analytics and AI solutions. Drive automation and establish robust evaluation methodologies for LLMs and AI-driven systems, ensuring alignment with business, regulatory, and ethical standards.

What you'd actually do

  1. Lead the end-to-end User Acceptance Testing (UAT), business validation, and evaluation processes for HR analytics products and enterprise AI solutions.
  2. Define and execute validation strategies that ensure AI models, predictive analytics frameworks, and data-driven solutions meet business, regulatory, and operational requirements.
  3. Drive the modernization and automation of validation capabilities by implementing scalable testing frameworks, automated regression suites, and AI evaluation methodologies.
  4. Establish robust approaches for evaluating Large Language Models (LLMs) and AI-driven solutions, including assessments of model quality, business relevance, fairness, bias mitigation, and data integrity.
  5. Partner closely with HR leaders, product managers, data scientists, and technology teams to ensure seamless alignment between business requirements and technical solutions.

Skills

Required

  • Proven experience leading validation, testing, analytics, risk, operations, or related functions within a complex, matrixed organization.
  • Strong understanding of enterprise AI capabilities, Large Language Models (LLMs), machine learning concepts, and emerging AI technologies.
  • Experience establishing, leading, or overseeing automated testing frameworks, validation programs, or business evaluation processes.
  • Demonstrated knowledge of software development lifecycles, data analytics ecosystems, and enterprise technology delivery methodologies.
  • Strong understanding of structured and unstructured data processing, analytics pipelines, and data quality principles.
  • Experience evaluating business outcomes, model performance, and operational effectiveness through data-driven methodologies.
  • Exceptional stakeholder management and influencing skills, with the ability to engage effectively across technical and non-technical audiences.
  • Demonstrated success leading teams, managing talent, and scaling operational capabilities through automation and process transformation.
  • Strong verbal and written communication skills with the ability to present complex information to executive stakeholders.
  • Experience managing data privacy, compliance, governance, and ethical considerations associated with sensitive workforce or enterprise data.

Nice to have

  • Advanced business degree (MBA or equivalent) or graduate-level qualification in analytics, data science, artificial intelligence, or a related discipline.
  • Bachelor's degree in Engineering, Computer Science, Information Technology, Analytics, or a related technical field.
  • Experience within financial services, banking, or other highly regulated industries.
  • Familiarity with AI evaluation frameworks, prompt testing methodologies, synthetic data generation techniques, and model benchmarking approaches.
  • Working knowledge of Python, SQL, automation testing tools, or technologies used to validate analytics platforms, data pipelines, and AI models.
  • Experience driving enterprise-scale transformation initiatives focused on automation, digital modernization, or AI adoption.

What the JD emphasized

  • AI Evaluation
  • evaluating Large Language Models (LLMs) and AI-driven solutions
  • fairness, bias mitigation
  • data privacy, governance, compliance, and ethical AI standards

Other signals

  • AI Evaluation Center of Excellence
  • leading the newly integrated User Acceptance Testing (UAT), Business Validation, and AI Evaluation function
  • oversee the end-to-end validation lifecycle for AI-powered products, predictive analytics, and HR technology solutions
  • driving the transformation of evaluation processes through automation
  • Establish robust approaches for evaluating Large Language Models (LLMs) and AI-driven solutions, including assessments of model quality, business relevance, fairness, bias mitigation, and data integrity.