Principal AI Engineer

Salesforce Salesforce · Enterprise · San Francisco, California - Palo Alto, Illinois - Chicago, New York - New York, Washington - Seattle, Washington - Bellevue, CA

Salesforce is seeking a Principal AI Engineer to build and maintain the core infrastructure, CI/CD pipelines, and platform services for their ML/AI platform, focusing on powering autonomous AI agents at enterprise scale. The role involves designing agent harness infrastructure, implementing agentic loop patterns, building automated pipelines for agent improvement, and creating sandboxed execution environments. A significant part of the role includes implementing evaluation frameworks, building eval datasets, instrumenting agent traces for observability, and optimizing agent quality, latency, and cost. The engineer will also automate CI/CD pipelines with evaluation gates and build developer self-service tools.

What you'd actually do

  1. Design and build agent harness infrastructure: the scaffolding that wraps LLM calls, manages tool use, handles retries, enforces policy, and feeds results back into iterative improvement loops.
  2. Implement evaluation frameworks for agent behavior using a combination of vendor , open source or in house built tools — covering task success, tool selection accuracy, trajectory evaluation, hallucination rates, latency, and cost
  3. Build and maintain eval datasets, golden trace libraries, and regression test suites that run automatically on every agent code change
  4. Build and optimize CI/CD pipelines (GitHub Actions, ArgoCD) that cover not just code deployment but agent evaluation gates — no agent ships without passing its eval suite
  5. Build internal tools and developer self-service interfaces that let ML engineers and data scientists iterate on agents without platform team involvement

Skills

Required

  • Platform Engineering
  • ML Infrastructure Engineering
  • Software Engineering
  • Agent harness infrastructure
  • Agentic loops
  • Tool orchestration
  • Structured output handling
  • Multi-turn conversation management
  • CI/CD pipelines
  • Evaluation frameworks
  • Observability tooling
  • Sandboxing
  • API security

Nice to have

  • LLM calls
  • Retry mechanisms
  • Policy enforcement
  • Iterative improvement loops
  • Memory management
  • Reusable platform primitives
  • Agent traces
  • Regression surfacing
  • Failure routing
  • Prompt improvement
  • Model improvement
  • Versioning
  • Rollout controls
  • Rollback mechanisms
  • Code execution isolation
  • File system access isolation
  • Tiered autonomy models
  • Human approval workflows
  • Infrastructure layer enforcement
  • Replay capabilities
  • Dry-run capabilities
  • Task success evaluation
  • Tool selection accuracy evaluation
  • Trajectory evaluation
  • Hallucination rate evaluation
  • Latency evaluation
  • Cost evaluation
  • Golden trace libraries
  • Regression test suites
  • LLM call instrumentation
  • Tool invocation instrumentation
  • Intermediate reasoning instrumentation
  • Final output instrumentation
  • Grafana
  • Agent quality metrics
  • Signal tracking
  • Continuous quality improvement
  • Latency improvement
  • Cost improvement
  • Prompt tuning
  • Tool calling optimization
  • Context engineering
  • Model selection optimization
  • Distillation
  • Fine-tuning (SFT, DPO, RLHF)
  • Curated trace data
  • A/B tests
  • Shadow deployments
  • Eval suite gating
  • Docker builds
  • Package builds
  • Security scanning
  • Agent integration tests
  • Self-healing CI patterns
  • Developer self-service interfaces
  • Platform vision
  • Scalability planning
  • Alerting
  • PagerDuty
  • Traditional platform health monitoring
  • Agent-specific signal monitoring
  • Error rate monitoring
  • Tool call failure monitoring
  • Eval score drift monitoring
  • Security best practices
  • Auditable traces
  • Access controls
  • Security reviews
  • Compliance for agent workloads
  • GitHub Actions
  • ArgoCD

What the JD emphasized

  • 9+ years as a Platform Engineer, ML Infrastructure Engineer, or Software Engineer
  • Demonstrated experience building agent harness infrastructure using agentic loops, tool orchestration, structured output handling, multi-turn conversation management

Other signals

  • building the next generation of our ML/AI platform that doesn't just support ML models, but powers autonomous AI agents at enterprise scale
  • designing the harnesses, sandboxes, and evaluation frameworks that let AI agents be developed, tested, and trusted in production
  • own the end-to-end lifecycle from agent experiment to production deployment
  • implement evaluation frameworks for agent behavior
  • build and optimize CI/CD pipelines (GitHub Actions, ArgoCD) that cover not just code deployment but agent evaluation gates