Principal Machine Learning Engineer, Accelerated Apache Spark

NVIDIA NVIDIA · Semiconductors · Santa Clara, CA

This role focuses on applying ML/AI to optimize and accelerate Apache Spark workloads on NVIDIA GPUs, involving performance prediction, adaptive systems, and developing AI agents for system issue resolution and optimization. The role requires significant experience in ML/DL solution design, productionization, and large-scale data processing platforms like Spark, with a focus on LLM/GenAI, reinforcement learning, and adaptive ML systems.

What you'd actually do

  1. Design and implement machine learning solutions for performance prediction and optimization of GPU accelerated enterprise Apache Spark workloads.
  2. Develop advanced algorithms and adaptive systems to continuously improve the performance of Apache Spark workloads on GPUs.
  3. Develop AI-based agents and tools to assist with fixing system issues and application optimization.
  4. Collaborate with key partners and customers on the deployment of complex machine learning solutions in various environments.
  5. Maintain deep domain expertise by knowing the latest published advances in ML systems and algorithms.

Skills

Required

  • BS, MS, or PhD or equivalent experience in Machine Learning, Data Science, Computer Science or a closely related field
  • 12+ years of professional experience in designing, implementing, and productionizing high-quality ML/DL solutions
  • 5+ experience as technical lead in ML model development
  • Proven hands-on experience (2+ years) with large-scale data processing platforms, such as Apache Spark
  • Proven ability to employ modern tooling and sound techniques for all aspects of crafting, deploying, and maintaining machine learning models
  • Excellent programming skills in Python and Python data science related libraries like numpy, pandas, scikit-learn, scipy, pytorch, and tensorflow
  • Deep experience with sophisticated ML methodologies, including LLM/GenAI, reinforcement learning, and adaptive, on-line ML systems
  • Strong expertise in feature engineering, feature importance assessment, and developing boosted tree model solutions (e.g., XGBoost)

Nice to have

  • Understanding of the internal workings and architecture related to Apache Spark
  • Familiarity with NVIDIA GPUs and CUDA
  • Experience coding in Scala, Java, and/or C++

What the JD emphasized

  • 12+ years of professional experience in designing, implementing, and productionizing high-quality ML/DL solutions
  • 5+ experience as technical lead in ML model development
  • Proven hands-on experience (2+ years) with large-scale data processing platforms, such as Apache Spark
  • Deep experience with sophisticated ML methodologies, including LLM/GenAI, reinforcement learning, and adaptive, on-line ML systems

Other signals

  • ML/AI methods to empower enterprises to migrate Spark workloads onto GPUs at scale
  • Design and implement machine learning solutions for performance prediction and optimization of GPU accelerated enterprise Apache Spark workloads
  • Develop AI-based agents and tools to assist with fixing system issues and application optimization