Psychometrician, Aws Training & Certification

Amazon Amazon · Big Tech · Herndon, VA · Applied Science

Psychometrician role at AWS Training & Certification focused on supporting a global certification and credentialing program. Responsibilities include exam development, operational psychometric analysis, and research projects using advanced techniques like ML, NLP, and GenAI. Requires programming skills in R or Python and experience with large-scale assessment programs. The role aims to ensure the validity, reliability, and accuracy of AWS certification exams.

What you'd actually do

  1. Perform and support the main psychometric aspects of exam development and operations, including but not limited to automated test assembly, item and test analyses, optimal item bank design, job task analysis, standard setting, quality assurance, and project planning.
  2. Conduct main aspects of psychometric analysis in operational work including performing item analysis using psychometric methods, building optimal test forms and pools via optimization techniques, analyzing and monitoring item bank health, setting pass standards via standard setting studies, and supporting Job Task Analysis (JTA) to define and refresh test blueprints.
  3. Conduct main aspects of psychometric analysis in developing and applying statistical and psychometric modeling to evaluate and ensure AWS certification exams’ validity, reliability, applicability, efficiency, and accuracy.
  4. Participate in research projects to improve existing operational processes and quality using advanced techniques such as Machine Learning (ML), statistical modeling, Natural Language Processing (NLP), Generative Artificial Intelligence (GenAI), etc.
  5. Develop automation code using R or Python for psychometric workflow pipeline and other tasks to improve operational efficiencies.

Skills

Required

  • PhD
  • Statistics
  • Psychometrics
  • Educational Measurement
  • Quantitative Psychology
  • Data Science
  • Industrial-Organizational (I/O) Psychology
  • large-scale education, licensure, or certification assessment programs
  • operational psychometric tasks on large-scale education, licensure, or certification assessment programs including item analysis, equating and scaling, item response theory, classical test theory, form and pool assembly, item bank health analysis, standard setting, and job task analysis
  • complex test designs such as linear-on-the-fly testing (LOFT), computerized adaptive testing (CAT)
  • machine learning (ML) or natural language processing (NLP)
  • Programming skills in at least one script-based programming language (R, Python)

Nice to have

  • Experience in testing for global audiences
  • History of conference presentations and publications
  • Experience with data forensics methods

What the JD emphasized

  • large-scale education, licensure, or certification assessment programs
  • operational psychometric tasks on large-scale education, licensure, or certification assessment programs including item analysis, equating and scaling, item response theory, classical test theory, form and pool assembly, item bank health analysis, standard setting, and job task analysis
  • machine learning (ML) or natural language processing (NLP)

Other signals

  • psychometrician
  • assessment programs
  • ML
  • NLP
  • GenAI