Machine Learning Engineer I

at Zendesk · Enterprise · Pune, India

This role focuses on evaluating and improving AI-driven support experiences by assessing model outputs for correctness, relevance, tone, safety, and reasoning. The Machine Learning Engineer I will refine evaluation rubrics, create datasets, analyze results to identify failure modes, and partner with cross-functional teams to translate findings into model improvements. The role requires strong attention to detail, clear communication, and practical experience with ML/NLP evaluation or data annotation.

What you'd actually do

  1. Evaluate AI-generated responses across support use cases (ticket replies, summaries, intent labels, agent recommendations) using structured rubrics for correctness, relevance, tone, safety, and reasoning.
  2. Apply and refine standardized evaluation rubrics and scoring guidelines; train and onboard annotators when needed to ensure consistent assessments.
  3. Write clear, actionable feedback on model outputs that calls out errors, hallucinations, missing context, bias, or policy risks, and suggests concrete remediation steps.
  4. Create and maintain gold‑standard evaluation datasets, test cases, and regression suites used for model validation and release gating.
  5. Partner directly with ML engineers, researchers, product managers, and UX teams to refine evaluation criteria, define metrics, and translate findings into product priorities and model improvements.

Skills

Required

  • BS in Computer Science, Data Science, Statistics, Computational Linguistics or equivalent practical experience.
  • 2–5 years industry experience in ML/NLP evaluation, data annotation, or QA for language systems (internships and similar experience count).
  • Proficient in Python and data manipulation tools (pandas, numpy); experience with Git and basic data pipeline concepts.
  • Demonstrated ability to write clear, reproducible evaluation criteria and give concise, actionable feedback.
  • Strong attention to detail and the ability to spot subtle errors, inconsistencies, and unsafe outputs in natural language model responses.
  • Clear, empathetic written communication; able to summarize issues and propose improvement steps for multiple stakeholders (engineers, researchers, PMs).
  • Practical experience with annotation, dataset curation, or building evaluation suites for ML or NLP systems.

Nice to have

  • Familiarity with customer support workflows and the nuance of agent vs. end‑user messaging
  • A collaborative mindset: you enjoy partnering cross‑functionally to move findings into action and care about measurement and reproducibility.
  • Comfortable using Python and common ML tooling to run evaluations, analyze results, and maintain dataset artifacts.
  • Experience working with large language models (LLMs), prompt engineering, or fine‑tuning workflows.
  • Hands‑on experience building and maintaining gold‑standard datasets, synthetic test cases, or adversarial examples.
  • Familiarity with automated evaluation frameworks, A/B testing, and model monitoring/metrics.
  • Prior experience in customer support or SaaS product domains, understanding domain-specific safety/tone constraints.
  • Advanced degree (MS/PhD) in a relevant field or equivalent research experience is a plus.

What the JD emphasized

  • evaluate AI-driven support experiences
  • ensure our models produce correct, helpful, safe, and on‑brand responses
  • Evaluate AI-generated responses across support use cases
  • evaluation rubrics
  • model validation
  • model improvements
  • evaluate
  • evaluation criteria
  • model monitoring
  • evaluation frameworks
  • evaluation results

Other signals

  • evaluating and improving AI-driven support experiences
  • ensure our models produce correct, helpful, safe, and on‑brand responses
  • evaluate AI-generated responses across support use cases
  • partner directly with ML engineers, researchers, product managers, and UX teams to refine evaluation criteria, define metrics, and translate findings into product priorities and model improvements
  • analyze evaluation results to identify systemic failure modes and trends across model versions
Read full job description

Job Description

Join us at Zendesk, where we're on a mission to power exceptional service for every person on the planet. We're accelerating that ambition by building products rooted in AI, automation, and intelligent customer experiences, because behind every interaction lies an opportunity to make a human connection.

We’re looking for a Machine Learning Engineer I who will focus on evaluating and improving AI-driven support experiences. In this role, you’ll help ensure our models produce correct, helpful, safe, and on‑brand responses across ticket replies, summaries, intent detection, and recommendations—so Zendesk customers and their end users get reliable, trustable AI assistance.

What you get to do every day

  • Evaluate AI-generated responses across support use cases (ticket replies, summaries, intent labels, agent recommendations) using structured rubrics for correctness, relevance, tone, safety, and reasoning.
  • Apply and refine standardized evaluation rubrics and scoring guidelines; train and onboard annotators when needed to ensure consistent assessments.
  • Write clear, actionable feedback on model outputs that calls out errors, hallucinations, missing context, bias, or policy risks, and suggests concrete remediation steps.
  • Create and maintain gold‑standard evaluation datasets, test cases, and regression suites used for model validation and release gating.
  • Partner directly with ML engineers, researchers, product managers, and UX teams to refine evaluation criteria, define metrics, and translate findings into product priorities and model improvements.
  • Analyze evaluation results to identify systemic failure modes and trends across model versions, produce dashboards/reports, and recommend monitoring or retraining strategies.

What We’re Looking For

  • Strong attention to detail and the ability to spot subtle errors, inconsistencies, and unsafe outputs in natural language model responses.
  • Clear, empathetic written communication; able to summarize issues and propose improvement steps for multiple stakeholders (engineers, researchers, PMs).
  • Practical experience with annotation, dataset curation, or building evaluation suites for ML or NLP systems.
  • Familiarity with customer support workflows and the nuance of agent vs. end‑user messaging (preferred but not required).
  • A collaborative mindset: you enjoy partnering cross‑functionally to move findings into action and care about measurement and reproducibility.
  • Comfortable using Python and common ML tooling to run evaluations, analyze results, and maintain dataset artifacts.

Basic Qualifications

  • BS in Computer Science, Data Science, Statistics, Computational Linguistics or equivalent practical experience.
  • 2–5 years industry experience in ML/NLP evaluation, data annotation, or QA for language systems (internships and similar experience count).
  • Proficient in Python and data manipulation tools (pandas, numpy); experience with Git and basic data pipeline concepts.
  • Demonstrated ability to write clear, reproducible evaluation criteria and give concise, actionable feedback.

Preferred Qualifications

  • Experience working with large language models (LLMs), prompt engineering, or fine‑tuning workflows.
  • Hands‑on experience building and maintaining gold‑standard datasets, synthetic test cases, or adversarial examples.
  • Familiarity with automated evaluation frameworks, A/B testing, and model monitoring/metrics.
  • Prior experience in customer support or SaaS product domains, understanding domain-specific safety/tone constraints.
  • Advanced degree (MS/PhD) in a relevant field or equivalent research experience is a plus.

Location

Pune, India (on-site)

Please note that Zendesk can only hire candidates who are physically located and plan to work from Karnataka or Maharashtra. Please refer to the location posted on the requisition for where this role is based.

Hybrid: In this role, our hybrid experience is designed at the team level to give you a rich onsite experience packed with connection, collaboration, learning, and celebration - while also giving you flexibility to work remotely for part of the week. This role must attend our local office for part of the week. The specific in-office schedule is to be determined by the hiring manager.

The intelligent heart of customer experience

Zendesk software was built to bring a sense of calm to the chaotic world of customer service. Today we power billions of conversations with brands you know and love.

Zendesk believes in offering our people a fulfilling and inclusive experience. Our hybrid way of working, enables us to purposefully come together in person, at one of our many Zendesk offices around the world, to connect, collaborate and learn whilst also giving our people the flexibility to work remotely for part of the week.

As part of our commitment to fairness and transparency, we inform all applicants that artificial intelligence (AI) or automated decision systems may be used to screen or evaluate applications for this position, in accordance with Company guidelines and applicable law.

Zendesk is an equal opportunity employer, and we’re proud of our ongoing efforts to foster global diversity, equity, & inclusion in the workplace. Individuals seeking employment and employees at Zendesk are considered without regard to race, color, religion, national origin, age, sex, gender, gender identity, gender expression, sexual orientation, marital status, medical condition, ancestry, disability, military or veteran status, or any other characteristic protected by applicable law. We are an AA/EEO/Veterans/Disabled employer. If you are based in the United States and would like more information about your EEO rights under the law, please click here.

Zendesk endeavors to make reasonable accommodations for applicants with disabilities and disabled veterans pursuant to applicable federal and state law. If you are an individual with a disability and require a reasonable accommodation to submit this application, complete any pre-employment testing, or otherwise participate in the employee selection process, please send an e-mail to peopleandplaces@zendesk.com with your specific accommodation request.