-
Notifications
You must be signed in to change notification settings - Fork 328
Description
Describe the bug
When creating a TransformersModel
instance using the from_model
classmethod, the resulting object is missing several attributes (e.g.continuous_batching) that are normally initialized by the standard __init__
constructor. This leads to AttributeError exceptions during the evaluation pipeline, making the from_model
method unreliable for integrating pre-loaded models.
To Reproduce
1, In an evaluation script, pre-load a model from transformers.
2. Instantiate the lighteval model wrapper using this pre-loaded model: model = TransformersModel.from_model(model=hf_model, ...)
3. Pass this model instance to a lighteval.pipeline.Pipeline.
4. Run pipeline.evaluate().
5. The pipeline will crash with an AttributeError when an internal method attempts to access an attribute that was not initialized by from_model (for example, AttributeError: 'TransformersModel' object has no attribute 'continuous_batching')
.
Expected behavior
An object created via TransformersModel.from_model
should be fully and correctly initialized, possessing the exact same set of attributes as an object created via the standard TransformersModel(config=...)
constructor. It should be a drop-in replacement that works reliably within the pipeline.
Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.