Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
157 Open 1,610 Closed
157 Open 1,610 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Add BHS benchmark
#3265 opened Aug 25, 2025 by jmichaelov Loading…
1
Fix Afrobench MasakhanePOS
#3263 opened Aug 25, 2025 by Anri-Lombard Loading…
Adding SPaRC to lm eval harness
#3262 opened Aug 25, 2025 by lkaesberg Loading…
2
Add long-context evaluation benchmarks (LongBench v2, Babilong, InfiniteBench, Phonebook)
#3256 opened Aug 21, 2025 by Mariani-code Loading…
1
8
fix gsm8k normalization
#3254 opened Aug 20, 2025 by huaanrui Loading…
1
Main
#3250 opened Aug 20, 2025 by seongtaehong Loading…
3
Adding 3LM to lm eval harness
#3241 opened Aug 14, 2025 by GeorgeSherif Loading…
1
Trim thinking content from model output in IFEval
#3240 opened Aug 14, 2025 by davideguidobene Loading…
Remove gen_prefix space and add warning
#3239 opened Aug 14, 2025 by bendboaz Loading…
1
5
Adding support for evaluating with Mistral and Pixtral models
#3235 opened Aug 13, 2025 by LearnerSXH Loading…
1
2
Adding support for Structured Generation with XGrammar
#3232 opened Aug 12, 2025 by ceferisbarov Loading…
5 tasks
4
Pass dataset_kwargs for Unitxt tasks
#3230 opened Aug 11, 2025 by mprahl Loading…
Fewshot refactor
#3227 opened Aug 8, 2025 by baberabb Loading…
1 task
Fix the Unitxt init method to set the task name
#3225 opened Aug 8, 2025 by mprahl Loading…
1
Fix: respect target_delimiter when using a gen_prefix on multiple-choice tasks
#3220 opened Aug 7, 2025 by karanikolopoulos Loading…
1
Update openai_completions.py
#3215 opened Aug 7, 2025 by phseidl Loading…
2
add intel xpu support for HFLM
#3211 opened Aug 5, 2025 by kaixuanliu Loading…
4
Support for DDP+MP with native torch and no accelerate
#3205 opened Aug 3, 2025 by xgal Loading…
1
feat: COT trace response handling in evaluator and model classes
#3204 opened Aug 3, 2025 by hhh2210 Loading…
1
Add new task: kmmlu_pro, kmmlu_redux
#3198 opened Aug 1, 2025 by jeonghodot Loading…
3
Fixed #2552: Improve answer extraction for hendrycks_math
#3192 opened Jul 30, 2025 by JoonYong-Park Loading…
1
1
refactor registry
#3189 opened Jul 28, 2025 by baberabb Loading…
Leverage vllm's tokenizer_info endpoint to avoid manual duplication
#3185 opened Jul 25, 2025 by m-misiura Loading…
4
Remove generate_until (multiple_target and doc_to_choice indexing) logic in ConfigurableTask.process_results
#3169 opened Jul 21, 2025 by baberabb Loading…
Add eqbench tasks in Spanish and Catalan
#3168 opened Jul 21, 2025 by priverabsc Loading…
4
ProTip! Adding no:label will show everything without a label.