-
Notifications
You must be signed in to change notification settings - Fork 328
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
With a model max length at 10K on ifeval, I'm getting truncations on the input (with no input stored in the logs): most likely an issue with the truncation.
I suspect it's also linked to the refacto on loglikelihood: the padding no longer works for accelerate models.
To Reproduce
lighteval accelerate 'model_name=Qwen/Qwen3-32B,model_parallel=True,max_length=16384,generation_parameters={temperature:0.6,top_p:0.9,top_k:40,min_p:0,max_new_tokens:16384}' 'extended|ifeval|0|0' --save-details --output-dir tmp
Logs will contain empty input fields in ModelResponse, with around 400 truncation
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working