Reproducing auto-rubric results

[RAgNarokBear]

Thanks for maintaining this awesome project! It helps me a lot for my research in Reward Modeling.

In auto-rubric evaluation, I failed to reproduce the quantitative results reported in the paper.

So I wonder what is the prompt used in the official evaluation. Simply concatinating the Rubric Evaluation Prompt (Fig. 8) and Rubrics (Sec. G.1) as follows seems not working.

## Task Description
I will provide you with a set of rubrics, along with the current query and two responses. These rubrics are the primary basis for selecting the best answer. You must follow the steps specified in the Evaluation Process when conducting your evaluation process.

## Rubrics
## Theme 1: Factual Accuracy and Canonical Consistency
Theme: Ensure factual accuracy, canonical consistency, and avoid fabrication or hallucination in responses.
• Tip 1: For queries about Undertale, ensure all character motivations and gameplay mechanics align with established lore, avoiding speculative or contradictory claims.
• Tip 2: When discussing historical milestones like early synchronized sound cartoons, correctly attribute ”Steamboat Willie” instead of ”My Old Kentucky Home” to maintain reliability.
• Tip 3: In responses involving Hogwarts students, include only canonically portrayed students with academically accurate achievements, excluding professors or non-student figures.
• Tip 4: Avoid inventing Sumerian texts or fabricated survey links; instead, acknowledge missing context and request clarification when necessary, especially for niche cultural references.

## Theme 2: Strict Adherence to Prompt Requirements
Theme: Maintain strict adherence to prompt structure, formatting, and explicit user requirements.
• Tip 1: When asked for a single word, provide exactly one word without redundancy or additional suggestions, as in responses requiring minimal output.
• Tip 2: For prompts specifying 100 items, deliver a complete list even if the topic is broad, proactively selecting a relevant subject to fulfill the quantitative requirement.
• Tip 3: In tagline creation, directly incorporate core technology benefits like ”distance at impact” and avoid vague or redundant phrasing that dilutes product relevance.
• Tip 4: When the prompt requires the word ”scenery” followed by a colon and a one-word term, follow this exact syntactic structure without deviation.

## Theme 3: Clarity and Structured Organization
Theme: Prioritize clarity, conciseness, and structured organization to enhance readability and directness.
• Tip 1: For a ”Thank you” prompt, respond with a concise acknowledgment and an open invitation for further questions, avoiding assumptions about the user being a student or lawyer.
• Tip 2: When summarizing steps for building a dropshipping agent business, use bullet points or numbered lists to present key points logically and avoid hallucinated information.
• Tip 3: In audit findings related to deposit insurance boards, structure responses with precise, actionable items and conclude with a concise summary emphasizing implications.
• Tip 4: Avoid excessive formatting like bold text or unnecessary punctuation when explaining grammatical correctness, maintaining a straightforward and professional tone.

## Theme 4: Comprehensive and Detailed Analysis
Theme: Deliver comprehensive, detailed, and thematically coherent narratives or analyses that fully address all prompt elements.
• Tip 1: For a CFA Institute Investment Foundations® Certificate explanation, include curriculum, eligibility, exam format, preparation resources, benefits, and continuing education with specific examples.
• Tip 2: In a fantasy story response, incorporate rich narrative detail, distinct character development, and immersive world-building such as vivid settings and dynamic interactions.
• Tip 3: When addressing a tax-proportional legislature, outline mechanics, implications, data collection, representation quotas, equity concerns, and constitutional considerations comprehensively.
• Tip 4: For a horror anime scene, use INT./EXT. designations, emphasize atmospheric tension, and describe creature details like a rhombus tail and chameleon-like head to align with anime style.

## Theme 5: Narrative and Contextual Fidelity
Theme: Ensure narrative and contextual fidelity by preserving character dynamics, tone, and worldbuilding consistency.
• Tip 1: In responses involving Jade’s character, maintain her authoritative yet professional tone, avoiding hostile shifts that contradict established behavior.
• Tip 2: For stories featuring Emily from KikoRiki, preserve her role as a mischievous prankster and integrate the whimsical tone when describing her failed morph into Rosa and the orange rear end mishap.
• Tip 3: When continuing a narrative about diaper use over potty training, maintain a playful, child-friendly tone and avoid contradictions with the original theme.
• Tip 4: In therapeutic role-play scenarios, prioritize immersive engagement with the patient’s imaginative world through dialogue and validation, rather than clinical checklists.

## Process
1. Confirm the task scenario of the current query and select the corresponding evaluation rubrics.
2. Identify the best response that meets the most selected rubrics.

## Query
{query}

## Response A
{response_a}

## Response B
{response_b}

## Output Requirements
Please choose the better response. Response "A", "B", or "tie" within the tags. 
<preference>A/B/tie</preference>

[XieLipeng0830]

Thank you for the kind words and for using our project in your research.

Reproducing quantitative results in LLM-based evaluation can be sensitive to prompt structure and inference configurations. To help us identify why your results aren't matching the paper, could you provide a bit more detail on the following?

1. Benchmark & Dataset: Which specific benchmark or subset are you testing on? (e.g., RewardBench, RM-Bench, etc.)
2. Voting Strategy: In our experiments, we often employ a voting strategy (running multiple iterations and taking the majority) to reduce variance and improve robustness. If you are currently using a single-pass inference, the results may fluctuate compared to the reported numbers.
3. Inference Hyperparameters: Could you share your exact settings? Specifically, what Temperature are you using, and are there any other relevant hyperparameters (like top_p or max tokens) you’ve configured?
4. Model Version: Which specific model and checkpoint are you using as the judge?
5. Reasoning/Thinking Process: We typically find that results improve significantly when the model provides its reasoning before the final tag. In our implementation, we enable "Thinking Mode" when using Qwen models. For other non-reasoning models, we add a "Reason" section as a Chain-of-Thought (CoT) step before the final . Are you currently using a direct output or a reasoning-based approach?

Looking forward to your feedback so we can help you get closer to the paper's results!

[RAgNarokBear]

Thanks for reply.

Some settings in my evaluation:
Benchmark: RM-Bench
Voting: no voting (I noticed a voting@1 for RM-Bench in Sec. B)
Inference Hyperparameters: temp=0 & top_p=1 & max_tokens=32768
Model: Local Qwen3-8B model loaded with vllm.
Result: 80.81% (against 88.28% reported in Tab. 1)

Any idea to release the source code for auto-rubric evaluation?

[XieLipeng0830]

Hi! Thanks for the detailed information.

We apologize for any confusion caused by the current documentation and the difficulty in reproducing these specific numbers. We appreciate your patience as we work to make the repository more transparent.

The gap between 80.81% and 88.28% on RM-Bench is likely due to the specific evaluation protocol of this benchmark. Here are a few key points to clarify:

RM-Bench Evaluation Logic: RM-Bench is unique because it uses a 3x3 comparison matrix to evaluate the model's ability to distinguish between different response qualities. For each sample, there are 3 "chosen" and 3 "rejected" responses, forming 9 pairwise comparisons in total. Furthermore, the order of responses is randomly swapped (is_reverse) to eliminate position bias.

Prompt Purpose (Fig. 8 vs. Evaluation): It is important to note that the Rubric Evaluation Prompt (Fig. 8) in the paper is specifically designed for the Query-specific Rubric Generation phase (i.e., testing if a generated rubric can effectively distinguish between positive and negative samples).

Benchmark Wrappers: The actual evaluation code for a specific benchmark involves additional wrappers designed to handle the unique characteristics of different datasets. For example, these wrappers manage the 3x3 matrix logic for RM-Bench or the specific "tie" evaluation protocols used in RewardBench2. Without these wrappers, a direct prompt concatenation may not capture the full evaluation methodology.

Code Release Plan: We understand that both the prompt and the evaluation infrastructure are critical for reproduction. We have plans to supplement the "Validating Graders" module with the specific code used to evaluate these benchmarks. Please note that this release will specifically focus on our rubric-based evaluation methods, which differ from standard evaluation scripts. This will include the logic for RM-Bench, RewardBench2, and other protocols integrated with our rubric system. It may take some time to clean up and release, but it is a priority on our roadmap.

Thanks again for your patience and for helping us improve the project!