Asset Management - AI Systems Engineer – Associate/vp

JPMorgan Chase · Banking · Shanghai, China · Asset & Wealth Management

The role focuses on building and optimizing enterprise LLM serving platforms, including GPU pooling, AI infrastructure, and MLOps for model deployment. It requires expertise in Python, Java, Kubernetes, and LLM inference engines, with a strong emphasis on performance optimization.

What you'd actually do

Build and optimize enterprise LLM serving platforms (e.g., vLLM, TensorRT-LLM) using techniques like PagedAttention, continuous batching, and quantization (AWQ/FP8) for high throughput and low latency.
Design GPU pooling, virtualization, and scheduling solutions on Kubernetes to maximize hardware utilization.
Manage distributed training clusters and high-performance networking (RDMA/NCCL).
Streamline the CI/CD pipeline for AI models.
Implement automated benchmarking, zero-downtime deployment, and comprehensive observability (TTFT, TPS, GPU metrics).

Skills

Required

Python
Java
Linux
Distributed Systems
Kubernetes
Docker
LLM inference engines
GPU architecture
CUDA programming
Distributed training frameworks
High-performance networking

Nice to have

vLLM
TensorRT-LLM
TGI
PagedAttention
continuous batching
quantization
AWQ
FP8
GPU pooling
virtualization
scheduling
Kubernetes operators
custom K8s controllers
DeepSpeed
Megatron-LM
Ray
RDMA
NCCL
CI/CD
benchmarking
zero-downtime deployment
observability
TTFT
TPS
GPU metrics
backend systems
MLOps
open-source AI Infra projects
custom CUDA kernels
Triton
operator fusion

What the JD emphasized

LLM serving
GPU optimization
ML Systems
Python
Java
Linux internals
Kubernetes
LLM inference engines
GPU architecture
CUDA programming
distributed training frameworks
high-performance networking

Other signals

LLM serving platforms
GPU pooling
MLOps

Read full job description

Key Responsibilities

Inference Platform & Optimization: Build and optimize enterprise LLM serving platforms (e.g., vLLM, TensorRT-LLM) using techniques like PagedAttention, continuous batching, and quantization (AWQ/FP8) for high throughput and low latency.
GPU Pooling & AI Infra: Design GPU pooling, virtualization, and scheduling solutions on Kubernetes to maximize hardware utilization. Manage distributed training clusters and high-performance networking (RDMA/NCCL).
Model Deployment & MLOps: Streamline the CI/CD pipeline for AI models. Implement automated benchmarking, zero-downtime deployment, and comprehensive observability (TTFT, TPS, GPU metrics).

Qualifications

Education & Experience:

Bachelor’s, Master’s, or Ph.D. in Computer Science, Computer Engineering, or a related field.
3+ years of experience in Backend Systems, Distributed Systems, or AI Infrastructure/MLOps, with at least 1-2 years specifically focused on LLM serving, GPU optimization, or ML Systems.

Core Engineering & Systems Skills:

Expert-level proficiency in Python and strong proficiency in Java (essential for inference engines and CUDA integration).
Deep understanding of Linux internals, networking, and distributed systems architecture.
Hands-on experience with container orchestration (Kubernetes, Docker) and building custom K8s operators or controllers.

AI Infrastructure & Optimization Skills:

Deep familiarity with LLM inference engines (vLLM, TensorRT-LLM, TGI) and understanding of their underlying architectural designs. Or
Solid understanding of GPU architecture (NVIDIA Ampere/Hopper), CUDA programming, and GPU memory management. Or
Experience with distributed training frameworks (DeepSpeed, Megatron-LM, Ray) and high-performance networking (RDMA, RoCE, InfiniBand).

Mindset & Soft Skills:

A "hacker" mindset with a passion for squeezing every drop of performance out of hardware.
Ability to collaborate effectively with AI Researchers (to understand their models) and Backend Engineers (to integrate AI into business systems).

Preferred

Contributions to open-source AI Infra projects (e.g., vLLM, Ray, PyTorch).
Experience writing custom CUDA kernels or using Triton for operator fusion.
Financial industry (Asset Management/Quant) experience is a plus.
Language: Professional working proficiency in English to collaborate with global teams.

Key Responsibilities

Inference Platform & Optimization: Build and optimize enterprise LLM serving platforms (e.g., vLLM, TensorRT-LLM) using techniques like PagedAttention, continuous batching, and quantization (AWQ/FP8) for high throughput and low latency.
GPU Pooling & AI Infra: Design GPU pooling, virtualization, and scheduling solutions on Kubernetes to maximize hardware utilization. Manage distributed training clusters and high-performance networking (RDMA/NCCL).
Model Deployment & MLOps: Streamline the CI/CD pipeline for AI models. Implement automated benchmarking, zero-downtime deployment, and comprehensive observability (TTFT, TPS, GPU metrics).

Qualifications

Education & Experience:

Bachelor’s, Master’s, or Ph.D. in Computer Science, Computer Engineering, or a related field.
3+ years of experience in Backend Systems, Distributed Systems, or AI Infrastructure/MLOps, with at least 1-2 years specifically focused on LLM serving, GPU optimization, or ML Systems.

Core Engineering & Systems Skills:

Expert-level proficiency in Python and strong proficiency in Java (essential for inference engines and CUDA integration).
Deep understanding of Linux internals, networking, and distributed systems architecture.
Hands-on experience with container orchestration (Kubernetes, Docker) and building custom K8s operators or controllers.

AI Infrastructure & Optimization Skills:

Deep familiarity with LLM inference engines (vLLM, TensorRT-LLM, TGI) and understanding of their underlying architectural designs. Or
Solid understanding of GPU architecture (NVIDIA Ampere/Hopper), CUDA programming, and GPU memory management. Or
Experience with distributed training frameworks (DeepSpeed, Megatron-LM, Ray) and high-performance networking (RDMA, RoCE, InfiniBand).

Mindset & Soft Skills:

A "hacker" mindset with a passion for squeezing every drop of performance out of hardware.
Ability to collaborate effectively with AI Researchers (to understand their models) and Backend Engineers (to integrate AI into business systems).

Preferred

Contributions to open-source AI Infra projects (e.g., vLLM, Ray, PyTorch).
Experience writing custom CUDA kernels or using Triton for operator fusion.
Financial industry (Asset Management/Quant) experience is a plus.
Language: Professional working proficiency in English to collaborate with global teams.