Skip to content

[BUG] Logs are missing some fields, truncation likely buggy #892

@clefourrier

Description

@clefourrier

Describe the bug

With a model max length at 10K on ifeval, I'm getting truncations on the input (with no input stored in the logs): most likely an issue with the truncation.

I suspect it's also linked to the refacto on loglikelihood: the padding no longer works for accelerate models.

To Reproduce

lighteval accelerate 'model_name=Qwen/Qwen3-32B,model_parallel=True,max_length=16384,generation_parameters={temperature:0.6,top_p:0.9,top_k:40,min_p:0,max_new_tokens:16384}' 'extended|ifeval|0|0' --save-details --output-dir tmp

Logs will contain empty input fields in ModelResponse, with around 400 truncation

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions