Software Engineer, Openshell

NVIDIA NVIDIA · Semiconductors · United States · Remote

NVIDIA OpenShell is seeking a Research Engineer to build the runtime infrastructure for secure, scalable, production-grade AI agents. The role involves translating cutting-edge agentic systems research into product features, validating impact through prototypes and benchmarks, and integrating new ideas into the OpenShell product. Responsibilities include tracking agent research (tool use, planning, memory, safety), bridging research and product, benchmarking methods, building prototypes, red-teaming systems, securing workflows, and collaborating across teams. Requires 8+ years of experience, an MS/PhD, and deep experience in LLMs, agent harnesses, multimodal models, evaluation frameworks, synthetic data, post-training, inference infrastructure, or adversarial ML. Experience with secure runtimes, compliance, LLM inference optimization (Triton, TensorRT-LLM, vLLM), or open-source AI development is a plus.

What you'd actually do

  1. Track the cutting edge: How are agentic systems evolving? You'll follow research in tool use, planning, memory, evaluation, self-improvement, multi-agent workflows, runtime infrastructure, and agent safety/security.
  2. Bridge research and product: Identify research ideas that can meaningfully improve OpenShell and translate them into concrete product opportunities.
  3. Benchmark and adapt: Reproduce and test promising methods from papers, open-source projects, industry work, and internal NVIDIA research.
  4. Build rapid prototypes: Create hands-on proof-of-concepts using OpenShell, including agent harnesses, evaluation loops, self-improving workflows, and runtime-native developer experiences.
  5. Red-team systems: Design evaluation and red-team harnesses that measure agent reliability, usefulness, scalability, safety, security, and developer experience.

Skills

Required

  • LLMs
  • agent harnesses
  • multimodal generative models
  • evaluation frameworks
  • synthetic data generation
  • post-training
  • inference infrastructure/optimization
  • adversarial ML
  • agent safety/security
  • turning complex research into reusable products, tools, demos, benchmarks, or production systems at scale
  • drive independent technical investigation
  • survey relevant work
  • run experiments
  • form a clear point of view
  • communicate findings clearly
  • strong product sense
  • care for UX and AX
  • focus on real-world impact
  • outstanding team orientation
  • comfort collaborating across research, engineering, product, design, solutions, and developer-facing teams

Nice to have

  • secure agent runtimes
  • tool sandboxing
  • capability-based security
  • enterprise policy systems
  • compliance or enterprise governance requirements such as auditability, data retention, access control, SOC2, HIPAA, GDPR, or regulated deployment environments
  • LLM inference infrastructure
  • model serving
  • inference optimization using tools such as Triton, TensorRT-LLM, vLLM, SGLang, Ray, Kubernetes, or cloud GPU platforms
  • integrating inference backends into agentic systems
  • routing across models
  • tool-aware context management
  • streaming
  • structured outputs
  • retries
  • monitoring
  • cost/performance optimization
  • developing or maintaining open-source software in AI agents, LLM systems, developer tooling, ML infrastructure, model serving, or related areas

What the JD emphasized

  • 8+ years of professional practical experience
  • MS/PhD in Computer Science, Physics, or a related field or equivalent experience
  • Deep experience in several of the following: LLMs, agent harnesses, multimodal generative models, evaluation frameworks, synthetic data generation, post-training, inference infrastructure/optimization, adversarial ML, or agent safety/security.

Other signals

  • building runtime infrastructure for secure, scalable, production-grade AI agents
  • translate the latest advances in agentic systems into runtime, tools, and workflows
  • identify promising methods from academia, industry, open source, and internal NVIDIA research
  • validate their impact through hands-on prototypes, benchmarks, and real agent workflows
  • integrate the best ideas into the product