Senior Data Scientist, AI & Model Risk

Block Block · Fintech · CA · Remote · 30315 Foundational - SFS - Ops

Senior Data Scientist role focused on AI & Model Risk, leading risk assessments for Generative AI and LLM use cases in financial services. The role involves ensuring safe, compliant, and responsible deployment by evaluating system design, testing, monitoring, and regulatory compliance. It requires strong understanding of AI architectures, testing methodologies, and risk management principles, with a focus on operationalizing AI governance.

What you'd actually do

  1. Lead end-to-end AI Risk Assessments for generative AI and LLM use cases across the Bank; Embedding in Block's enterprise-wide GenAI review process, coordinating cross-functional SMEs (Legal, Compliance, InfoSec, Data Governance, MRM, ERM, BRC, TPRM, Financial Crimes), and managing timelines to ensure reviews are completed within SLA.
  2. Review AI system design and documentation; Including retrieval sources, assumptions, limitations, fallback plans, guardrail configurations, and change management procedures — across banking use cases such as fraud detection, BSA/AML compliance, credit decisioning, and customer-facing applications, ensuring governance controls are commensurate with each use case's risk profile.
  3. Assess pre-deployment testing for adequacy inclusive of output integrity, hallucination detection, boundary and edge case testing, ethical and safety guardrails, bias testing, A/B testing, volume testing, and UAT — designing and conducting independent testing as needed.
  4. Evaluate ongoing monitoring plans for comprehensiveness - including accuracy, hallucination rates, drift detection, sensitive data controls, reliability metrics, CSAT, acceptable performance ranges, and documented remediation procedures.
  5. Develop and maintain templates, tools, and procedures to support the effectiveness and scalability of the AI Risk Governance Program.

Skills

Required

  • 5 years of related experience with a Bachelor’s degree in a quantitative field; or 3 years and a Master’s degree; or a PhD without experience; or equivalent work experience in risk management, model risk management, or AI risk management
  • Proficiency in Python or similar languages for evaluating AI system behavior, writing test scripts, or analyzing model outputs
  • Strong understanding of generative AI architectures; Including LLMs, transformer models, RAG systems, and agentic AI, plus hands-on experience interacting with and critically evaluating these systems, sufficient to assess design decisions, output quality, and limitations
  • Understanding of interagency model risk management principles, including SR 26-2
  • Knowledge of AI testing methodologies, ex. functional testing, bias testing, adversarial testing, and performance monitoring plus familiarity with data privacy and security principles (encryption, access controls, data classification)
  • Excellent written and verbal communication and the ability to translate complex technical AI concepts for non-technical stakeholders, senior management, and regulators
  • Strong analytical judgment with the ability to manage multiple concurrent assessments, prioritize effectively, and drive risk-based decisions with minimal day-to-day oversight

Nice to have

  • Master's degree in AI/ML, Cybersecurity, Data Science, or related field
  • Familiarity with AI governance frameworks (NIST AI RMF, ISO 42001, or equivalent) and the FFIEC IT Examination Handbooks
  • Experience with AI governance tools and platforms (model registries, monitoring dashboards, risk scoring systems)
  • Experience with explainability tools (SHAP, LIME, attention visualization)
  • Certifications: CRISC, PRM, FRM, or AI-specific certifications such as NIST AI RMF practitioner or ISO 42001 Lead Implementer
  • Prior experience in a second-line-of-defense or internal audit role at a bank or financial institution
  • Experience developing AI risk governance frameworks in environments where prescriptive regulatory guidance does not yet exist

What the JD emphasized

  • lead and coordinate AI risk assessments
  • Model Risk Management principles
  • safe, compliant, and responsible deployment
  • AI Governance
  • Information Security, Compliance, and Legal stakeholders
  • complex and ambiguous challenges
  • sound judgment
  • risk-based decisions
  • senior stakeholders
  • strengthen governance practice
  • shaping how AI risk management works
  • Operationalizing principles into processes
  • safe, effective AI deployment
  • shaping how AI gets deployed safely and responsibly
  • end-to-end AI Risk Assessments
  • generative AI and LLM use cases
  • enterprise-wide GenAI review process
  • cross-functional SMEs
  • ensure reviews are completed within SLA
  • Assess pre-deployment testing
  • output integrity
  • hallucination detection
  • boundary and edge case testing
  • ethical and safety guardrails
  • bias testing
  • A/B testing
  • volume testing
  • UAT
  • independent testing
  • Evaluate ongoing monitoring plans
  • accuracy
  • hallucination rates
  • drift detection
  • sensitive data controls
  • reliability metrics
  • CSAT
  • acceptable performance ranges
  • documented remediation procedures
  • templates, tools, and procedures
  • effectiveness and scalability
  • AI Risk Governance Program
  • evolving regulatory landscape
  • AI in banking
  • FFIEC IT Examination Handbook standards
  • FDIC Financial Institution Letters
  • interagency statements
  • SR 26-2
  • emerging guidance
  • regulatory inquiries
  • minimum of 5 years of related experience
  • Bachelor’s degree in a quantitative field
  • 3 years and a Master’s degree
  • PhD without experience
  • equivalent work experience in risk management, model risk management, or AI risk management
  • Proficiency in Python or similar languages
  • evaluating AI system behavior
  • writing test scripts
  • analyzing model outputs
  • Strong understanding of generative AI architectures
  • LLMs
  • transformer models
  • RAG systems
  • agentic AI
  • hands-on experience interacting with and critically evaluating these systems
  • assess design decisions
  • output quality
  • limitations
  • Understanding of interagency model risk management principles
  • Knowledge of AI testing methodologies
  • functional testing
  • bias testing
  • adversarial testing
  • performance monitoring
  • data privacy and security principles
  • encryption
  • access controls
  • data classification
  • Excellent written and verbal communication
  • translate complex technical AI concepts for non-technical stakeholders, senior management, and regulators
  • Strong analytical judgment
  • manage multiple concurrent assessments
  • prioritize effectively
  • drive risk-based decisions
  • minimal day-to-day oversight
  • Master's degree in AI/ML, Cybersecurity, Data Science, or related field
  • AI governance frameworks
  • NIST AI RMF
  • ISO 42001
  • FFIEC IT Examination Handbooks
  • AI governance tools and platforms
  • model registries
  • monitoring dashboards
  • risk scoring systems
  • explainability tools
  • SHAP
  • LIME
  • attention visualization
  • Certifications: CRISC, PRM, FRM
  • AI-specific certifications
  • NIST AI RMF practitioner
  • ISO 42001 Lead Implementer
  • second-line-of-defense
  • internal audit role at a bank or financial institution
  • developing AI risk governance frameworks
  • prescriptive regulatory guidance does not yet exist

Other signals

  • AI Risk Assessments
  • Generative AI
  • LLM use cases
  • Model Risk Management
  • AI Governance
  • regulatory landscape