This directory contains a comprehensive evaluation framework for assessing Large Language Model (LLM) performance on diverse question types and answer formats. The system supports multiple data types and comparison methods to ensure accurate evaluation across various problem domains.
The evaluation framework is designed to handle complex answer formats including:
- Multiple choice questions with single or multiple correct answers
- Numerical computations and mathematical expressions
- Text/nominal answers with normalization
- Ordered and unordered arrays
- Multi-dimensional data structures
- Mixed-type responses
- LaTeX mathematical notation
- 🔍 Flexible Answer Extraction: Extracts answers from LaTeX
\boxed{}notation and various text formats - 🎯 Multi-Type Evaluation: Supports different evaluation strategies based on answer type with automatic detection
- 🛠️ Robust Parsing: Handles JSON, LaTeX tables, malformed data, and mixed text formats
- 📊 Score Computation: Provides both binary evaluation and partial scoring for multi-part questions
- 🌍 Language Support: Handles both English and Chinese text, including number word conversion
- ⚡ Performance Optimized: Includes timeout handling and concurrent evaluation for large-scale assessments
git clone https://github.com/puzzleclone/PolyhedronEvaluator.git
pip install -r requirements.txtfrom main import compute_score, evaluation
# Simple evaluation for binary correctness
prediction = "\\boxed{A}"
reference = "A"
result = evaluation(prediction, reference, "option")
print(result) # True
# Numerical evaluation
result = evaluation("\\boxed{25}", "twenty-five", "numeral")
print(result) # True
# Array evaluation
pred_array = "\\boxed{['apple', 'banana', 'cherry']}"
ref_array = "['apple', 'banana', 'cherry']"
result = evaluation(pred_array, ref_array, "ordered array")
print(result) # True# Multi-part question scoring
solution = "\\boxed{A}\\boxed{100}"
ground_truth = "A====125"
eval_types = "option,numeral"
score = compute_score(solution, ground_truth, eval_types)
print(f"Score: {score}") # 0.5 (1 answer out of 2 is correct)# Complex array evaluation
solution_str = "\\boxed{[['a', 'b'], ['c', 'd']]}"
ground_truth = "[['b', 'a'], ['d', 'c']]"
score = compute_score(solution_str, ground_truth, "oua_nominal")
print(f"Ordered outer, unordered inner array score: {score}") # 1.0
# Multiple choice with multiple answers
result = evaluation("\\boxed{A, B, C}", "ABC", "multi_answers_MCQ")
print(result) # True"option": Multiple choice answers (A, B, C, etc.)"nominal": Text-based answers with normalization"numeral": Numerical computations and expressions
"ordered array": Sequence-sensitive lists"unordered array": Order-independent sets"subset": Subset relationship validation
"multi_answers_MCQ": Multiple correct options in multiple choice"ooa_nominal": Ordered outer, ordered inner arrays (nominal data)"oua_numeral": Ordered outer, unordered inner arrays (numerical data)"uoa_nominal": Unordered outer, ordered inner arrays- And other complex multi-dimensional combinations
PolyhedronEvaluator/
├── main.py # Main evaluation logic and routing
├── extract_answer.py # Answer extraction from model outputs
├── eval_multiple_choice.py # Multiple choice question evaluation
├── eval_nominal.py # Text/name-based answer evaluation
├── eval_numeral.py # Numerical answer evaluation with SymPy
├── eval_array.py # Array and sequence evaluation
├── eval_utils.py # Utility functions and text processing
└── README.md # This documentation
Scoring function for reinforcement learning training with partial credit support.
Parameters:
solution_str: Complete model output with potential multiple answersground_truth: Ground truth with potential multiple partseval_type_text: Comma-separated evaluation types
Returns: Float score between 0.0 and 1.0 (suitable as RL reward)
Strict binary evaluation for model inference assessment.
Returns: True if all parts are correct, False otherwise
- 📝 Number Word Conversion: Converts English ("twenty-five") and Chinese ("二十五") number words to digits
- 🔧 Punctuation Normalization: Removes and normalizes punctuation across languages
- 📊 LaTeX Table Parsing: Extracts structured data from LaTeX table formats
- 🛠️ JSON Repair: Handles malformed JSON with automatic correction
- 🧮 Mathematical Expression Handling: Processes complex mathematical notation
- Plain text answers
- LaTeX mathematical expressions with
\boxed{}notation - JSON arrays and objects
- LaTeX tables (
\begin{array},\begin{bmatrix}) - Mixed format responses with multiple
\boxed{}sections - Comma/semicolon separated lists
- Multi-line responses
The system includes robust error handling for:
- ✅ Malformed JSON structures
- ✅ Invalid LaTeX expressions
- ✅ Missing or incomplete answers
- ✅ Type conversion errors
- ✅ Parsing failures
- ✅ Timeout protection for complex symbolic computation
All evaluation functions return appropriate defaults (False/0.0) when errors occur, ensuring the evaluation pipeline continues running.
- ⏱️ Timeout Protection: Symbolic computations are protected by timeouts to prevent hanging
- 🔄 Concurrent Processing: Uses ThreadPoolExecutor for performance-critical operations
- 🎯 Efficient Parsing: Optimized parsing strategies for different data formats
- 📊 Batch Processing: Designed to handle large-scale evaluation efficiently
When adding new evaluation types:
- Implement the evaluation logic in the appropriate
eval_*.pyfile - Add the type routing in
main.py's internal routing function - Update the documentation with the new type
- Add comprehensive test cases to verify functionality
- Ensure proper error handling and edge case coverage
This evaluation framework is part of the PuzzleClone project and is licensed under the Apache 2.0 License.
Our evaluation code is partially modified from LIMO. We thank their team for their valuable contributions to the community.