Docs/sample reports #39
Conversation
…esults from git - Move oncology_translation_report.md to docs/applications/examples/ - Add reference to example report in zero_shot_evaluation.md - Add /evaluation_results to .gitignore - Fix examples/ gitignore rule to only ignore root-level examples folder
- Regenerate evaluation report in English using config.yaml - Update zero_shot_evaluation.md reference text to English - Replace Chinese example report with English version
- Consolidate 'When to Use' and 'How It Works' into concise 'Overview' - Reorganize advanced topics into tabbed sections for better readability - Simplify configuration examples and reduce redundant explanations - Replace verbose 'Next Steps' with compact 'Related Topics' links
Summary of ChangesHello @XiaoBoAI, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the zero-shot evaluation framework by adding detailed configuration examples and a real-world sample report, making it easier for users to understand and implement. It also refines the associated documentation for better clarity and introduces foundational components for future local model integration. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive zero-shot evaluation pipeline, including new configuration examples (config.yaml, minimal_config.yaml) and a sample evaluation report (oncology_translation_report.md). It also adds a new base configuration for local model providers (base_config.py) to openjudge/models/local. The documentation for zero-shot evaluation (zero_shot_evaluation.md) has been significantly restructured for clarity, moving from a step-by-step guide to a component-based guide with tabbed sections for quick start and advanced topics, and including details on report generation. A review comment highlighted an inconsistency in the documentation's 'Component Guide', noting that it listed four steps while the overall process is five, and suggested clarifying the role of ZeroShotPipeline as an orchestrator by detailing five distinct steps with specific components like GradingRunner and PairwiseAnalyzer.
| <div class="workflow-single"> | ||
| <div class="workflow-header">Pipeline Components</div> | ||
|
|
||
| # Configure task and endpoint | ||
| task = TaskConfig( | ||
| description="Code review assistant for Python", | ||
| scenario="Review code for bugs, style issues, and improvements" | ||
| ) | ||
| <div class="workflow"> | ||
| <ol class="workflow-steps"> | ||
| <li><strong>Generate Test Queries</strong> | ||
|
|
||
| judge_endpoint = OpenAIEndpoint( | ||
| base_url="https://api.openai.com/v1", | ||
| api_key="your-api-key", | ||
| model="gpt-4" | ||
| ) | ||
| Use `QueryGenerator` to create diverse test queries from your task description. Supports parallel generation, automatic deduplication, and optional Evol-Instruct complexity evolution.</li> | ||
| <li><strong>Collect Responses</strong> | ||
|
|
||
| query_config = QueryGenerationConfig( | ||
| num_queries=20, | ||
| seed_queries=["Review this Python function for bugs..."], | ||
| enable_evolution=True, # Enable Evol-Instruct | ||
| evolution_rounds=1 | ||
| ) | ||
| Use `ResponseCollector` to query all target models concurrently and gather their responses for comparison.</li> | ||
| <li><strong>Generate Evaluation Rubrics</strong> | ||
|
|
||
| generator = QueryGenerator(judge_endpoint, task, query_config) | ||
| queries = await generator.generate() | ||
| ``` | ||
| Use `TaskBasedRubricGenerator` to automatically create evaluation criteria (accuracy, completeness, clarity, etc.) tailored to your specific task.</li> | ||
| <li><strong>Run Pairwise Evaluation</strong> | ||
|
|
||
| !!! info "Query Generation Features" | ||
| - **Parallel Batches**: Generates queries in parallel for diversity | ||
| - **Deduplication**: Automatically removes duplicate/similar queries | ||
| - **Evol-Instruct**: Optional complexity evolution for harder queries | ||
| - **Category Balancing**: Balance queries across specified categories | ||
| Use `ZeroShotPipeline` to orchestrate the full evaluation, comparing all response pairs and producing final rankings.</li> | ||
| </ol> | ||
| </div> | ||
| </div> |
There was a problem hiding this comment.
This component guide is a bit confusing as it only lists four steps, which is inconsistent with the five-step process described in the text and the table above. It also misrepresents the role of ZeroShotPipeline.
To improve clarity and accuracy, I suggest restructuring this into five distinct steps, each mapping to a specific component from the codebase. This will give readers a clearer understanding of the pipeline's architecture. I've also added a note to clarify that ZeroShotPipeline is the high-level orchestrator that ties everything together.
| <div class="workflow-single"> | |
| <div class="workflow-header">Pipeline Components</div> | |
| # Configure task and endpoint | |
| task = TaskConfig( | |
| description="Code review assistant for Python", | |
| scenario="Review code for bugs, style issues, and improvements" | |
| ) | |
| <div class="workflow"> | |
| <ol class="workflow-steps"> | |
| <li><strong>Generate Test Queries</strong> | |
| judge_endpoint = OpenAIEndpoint( | |
| base_url="https://api.openai.com/v1", | |
| api_key="your-api-key", | |
| model="gpt-4" | |
| ) | |
| Use `QueryGenerator` to create diverse test queries from your task description. Supports parallel generation, automatic deduplication, and optional Evol-Instruct complexity evolution.</li> | |
| <li><strong>Collect Responses</strong> | |
| query_config = QueryGenerationConfig( | |
| num_queries=20, | |
| seed_queries=["Review this Python function for bugs..."], | |
| enable_evolution=True, # Enable Evol-Instruct | |
| evolution_rounds=1 | |
| ) | |
| Use `ResponseCollector` to query all target models concurrently and gather their responses for comparison.</li> | |
| <li><strong>Generate Evaluation Rubrics</strong> | |
| generator = QueryGenerator(judge_endpoint, task, query_config) | |
| queries = await generator.generate() | |
| ``` | |
| Use `TaskBasedRubricGenerator` to automatically create evaluation criteria (accuracy, completeness, clarity, etc.) tailored to your specific task.</li> | |
| <li><strong>Run Pairwise Evaluation</strong> | |
| !!! info "Query Generation Features" | |
| - **Parallel Batches**: Generates queries in parallel for diversity | |
| - **Deduplication**: Automatically removes duplicate/similar queries | |
| - **Evol-Instruct**: Optional complexity evolution for harder queries | |
| - **Category Balancing**: Balance queries across specified categories | |
| Use `ZeroShotPipeline` to orchestrate the full evaluation, comparing all response pairs and producing final rankings.</li> | |
| </ol> | |
| </div> | |
| </div> | |
| <div class="workflow-single"> | |
| <div class="workflow-header">Pipeline Components</div> | |
| <div class="workflow"> | |
| <ol class="workflow-steps"> | |
| <li><strong>Generate Test Queries</strong> | |
| Use `QueryGenerator` to create diverse test queries from your task description. Supports parallel generation, automatic deduplication, and optional Evol-Instruct complexity evolution.</li> | |
| <li><strong>Collect Responses</strong> | |
| Use `ResponseCollector` to query all target models concurrently and gather their responses for comparison.</li> | |
| <li><strong>Generate Evaluation Rubrics</strong> | |
| Use `TaskBasedRubricGenerator` to automatically create evaluation criteria (accuracy, completeness, clarity, etc.) tailored to your specific task.</li> | |
| <li><strong>Run Pairwise Comparisons</strong> | |
| Use `GradingRunner` to run pairwise comparisons between model responses using a judge model.</li> | |
| <li><strong>Analyze Results</strong> | |
| Use `PairwiseAnalyzer` to analyze the comparison results, calculate win rates, and produce final rankings.</li> | |
| </ol> | |
| </div> | |
| <p><em>Note: The <code>ZeroShotPipeline</code> class orchestrates all these components to provide a simple, end-to-end evaluation workflow.</em></p> | |
| </div> |
OpenJudge Version
[The version of OpenJudge you are working on, e.g.
import openjudge; print(openjudge.__version__)]Description
[Please describe the background, purpose, changes made, and how to test this PR]
Checklist
Please check the following items before code is ready to be reviewed.
pre-commit run --all-filescommand