Fine tuning qwen2.5 error[bug]

[shaojun0]

I want to fine-tune qwen2.5-3b, but there is an error in NPU:

I copied the code and executed it in CPU:

it was successfully executed:

I am trying to edit this file,print it：

But printing the statement also reports an error:

code:https://github.com/shaojun0/quen2_5_train
runtime environment：quay.io/ascend/vllm-ascend:v0.9.2rc1

[yunyiyun]

Currently, conv3d cannot fully support it
https://www.hiascend.com/document/detail/zh/Pytorch/700/apiref/apilist/ptaoplist_001090.html

You can try configuring the following environment variables
torch.npu.config.allow_internal_format = False
torch.npu.set_compile_mode(jit_compile=False)

[shaojun0]

Currently, conv3d cannot fully support it https://www.hiascend.com/document/detail/zh/Pytorch/700/apiref/apilist/ptaoplist_001090.html

You can try configuring the following environment variables torch.npu.config.allow_internal_format = False torch.npu.set_compile_mode(jit_compile=False)

I understand what you mean, but this doesn't seem to involve conv3d, so why is it still reporting this error?

[yunyiyun]

Currently, conv3d cannot fully support it https://www.hiascend.com/document/detail/zh/Pytorch/700/apiref/apilist/ptaoplist_001090.html
You can try configuring the following environment variables torch.npu.config.allow_internal_format = False torch.npu.set_compile_mode(jit_compile=False)

I understand what you mean, but this doesn't seem to involve conv3d, so why is it still reporting this error?

you cann see this prompt：
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1