Software Engineer, AI

Lattice Lattice · Enterprise · Remote · Engineering

Software Engineer, AI role focused on building and improving AI evaluation pipelines, production tracing, feedback systems, and AI system components like RAG and multi-step workflows. The role involves writing Python code, integrating with LLM providers, and supporting system reliability and observability. Experience with LLM concepts, backend systems, and cloud environments is required, with a strong interest in LLM-powered products.

What you'd actually do

  1. Contribute to AI evaluation pipelines, including offline evals, production tracing, and feedback systems.
  2. Implement and maintain performance metrics (e.g., response quality, task success, reliability) using established frameworks.
  3. Help create and maintain evaluation datasets and test cases to identify regressions.
  4. Analyze results and propose incremental improvements to model and agent quality.
  5. Contribute to AI system components such as RAG pipelines, retrieval systems, and multi-step workflows within existing architectures.

Skills

Required

  • Python
  • LLM concepts
  • RAG
  • evaluation
  • backend systems
  • APIs
  • cloud environments
  • logging
  • monitoring
  • debugging tools

Nice to have

  • LLMs
  • prompt iteration
  • MLOps
  • vector databases
  • retrieval systems
  • experimentation
  • metrics
  • statistical analysis
  • TypeScript
  • LangGraph
  • evaluation platforms

What the JD emphasized

  • 2–5 years of professional software engineering experience
  • Experience contributing to production systems as part of a team
  • Strong understanding of LLM concepts (prompting, RAG, evaluation)
  • Familiarity with backend systems, APIs, and cloud environments (e.g., AWS, GCP)
  • Exposure to logging, monitoring, or debugging tools

Other signals

  • AI Evaluation & Quality Systems
  • AI Systems & Infrastructure
  • LLM concepts (prompting, RAG, evaluation)
  • Python
  • backend systems, APIs, and cloud environments