Skip to content

[BUG] TransformersModel.from_model provides an incompletely initialized object compared to __init__ #897

@huaanrui

Description

@huaanrui

Describe the bug

When creating a TransformersModel instance using the from_model classmethod, the resulting object is missing several attributes (e.g.continuous_batching) that are normally initialized by the standard __init__ constructor. This leads to AttributeError exceptions during the evaluation pipeline, making the from_model method unreliable for integrating pre-loaded models.

To Reproduce

1, In an evaluation script, pre-load a model from transformers.
2. Instantiate the lighteval model wrapper using this pre-loaded model: model = TransformersModel.from_model(model=hf_model, ...)
3. Pass this model instance to a lighteval.pipeline.Pipeline.
4. Run pipeline.evaluate().
5. The pipeline will crash with an AttributeError when an internal method attempts to access an attribute that was not initialized by from_model (for example, AttributeError: 'TransformersModel' object has no attribute 'continuous_batching').

Expected behavior

An object created via TransformersModel.from_model should be fully and correctly initialized, possessing the exact same set of attributes as an object created via the standard TransformersModel(config=...) constructor. It should be a drop-in replacement that works reliably within the pipeline.

Version info

Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions