Content Engineering is a horizontal function in the Muse Product Post Training org within Meta Superintelligence Labs (MSL) that shapes AI product experiences by aligning models to be helpful in new product experience through productizing prompt engineering, frontier evaluations, and quality frameworks. Sitting at the intersection of user experience and large-language model behavior, content engineers partner closely with research science, engineering, product and design teams to build and ship AI experiences from lab to production, across modalities and surfaces. Great Content Engineers have the aesthetic taste to notice what makes a model output great, the technical prompting expertise to align models to consistently provide great responses, experience at building auto-gradeable frontier evals for new capabilities, and the experience delivering high scale datasets via human annotators and synthetic data. This role goes beyond shaping how the model communicates — it owns the quality bar. This person defines what "great" looks like across product capabilities, builds the evaluation infrastructure (human and automated) to measure it, and runs the feedback loops that turn qualitative insight into model improvement. They operate as a cross-functional bridge between research science, engineering, product, and policy, translating user-facing quality problems into structured priorities and actionable fixes. They serve as a point of contact for internal stakeholders across AI on priority product and model initiatives, and aim to build experiences that entertain, inform, and delight.
Responsibilities
Quality Definition & Frameworks: Define what "great" looks like for AI product capabilities — build guidelines, golden response sets, and frontier evals that set the quality bar across features and surfaces. Develop failure mode taxonomies that give engineering teams a structured, prioritized view of what's breaking and why. Evaluation & Measurement: Own the full human evaluation pipeline — design rubrics, guide contractor annotator teams, build calibration processes, and deliver data analysis on results. Partner with Research Science to build and validate auto-judges aligned with human raters; define the methodology for measuring alignment drift. Construct and run large-scale evals to track quality metrics across product capabilities over time. Testing & Iteration: Lead structured dogfooding and testing programs — design test plans targeting specific failure modes, run testing rounds, triage and categorize results, and deliver prioritized summaries to product and engineering. Operate at speed to unblock fast iteration — turn around quality assessments quickly enough to inform the current dev cycle, not the next one. Identify emerging quality issues before they reach external users. Craft and tune system prompts and agentic behavior to support product vision and model outcomes. Cross-Functional Leadership: Serve as a key member of the cross-functional team across the product capabilities you work on— aligning engineering, research science, product, policy, and design on quality standards and priorities. Lead through influence and collaboration across teams; make quality legible and actionable for technical partners. Agentic workflows: Leverage AI-native tools to replace manual workflows with scalable, repeatable processes across evaluation design, data analysis, visual development and cross-functional comms. Continuously evaluate and adopt emerging tools that allow the team to operate at speed. This is a role where experimentation is key, and a willingness to experiment, learn new tools, and build solutions hands-on is valued.
Qualifications
Bachelor's degree or equivalent experience with 5+ years of experience in digital content strategy, user experience, technical writing, journalism, production, or related fields Experience designing and running human evaluation pipelines at scale — including annotator management, rubric design, golden set construction, and calibration Experience translating ambiguous "it doesn't feel right" feedback into structured, objective, fixable categories Experience defining and operationalizing subjective quality dimensions into measurable benchmarks Experience running structured software testing/qa programs — designing test plans, triaging results, and delivering actionable analysis to engineering teams fast enough to matter in the current dev cycle Experience making editorial and content quality decisions in a fast-paced environment, communicating complex technical concepts to cross-functional partners, and leading through influence across teams without direct reporting lines Experience with AI-native tooling (LLM-based development tools, annotation platforms, prototyping environments) and a bias toward using them to move faster Experience with LLM-as-judge development — building automated quality signals aligned with human judgment, and validating that alignment over time Experience working with product teams or programs (or other equivalent fields) from roadmapping through delivery Experience working in prompt engineering and agentic workflows