Qwen-MOE finetune results in mixed language tokens (language switching in generated answers)

[chapter544]

Hi,
When I use megatron-swift to LORA finetune Qwen3-MOE-30B-3AB, upon converting the merged mcore back to huggingface format, the model tends to generate mixed languages tokens in the answer. For example, when I expect an English answer, then most of the tokens are in English, but there are also some Chinese tokens generated. The same observation is true if I expect a Chinese answer, I also see English and other languages' tokens mixed in the answer. If I use the google translate to translate these mixed tokens, they do have the "correct" in the meaning, but just in different languages.

For the LORA training, I just finetuned the linear-layers, NO training on the embedding and output layer. Flags that I used in the training script:

• method: SFT
• packing = true
• padding_free = true

I have a few thoughts in mind, but I wonder if you have any insights:

1. Could the "packing=true" is the issue?
2. Conversations between mcore and huggingface format??
3. Should I enable training the embedding and the output layers in order to minimize the language-switching issue?

Thanks.

[Jintao-Huang]

Please send the training script.

[chapter544]

Here is the training script:

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
NPROC_PER_NODE=2
CUDA_VISIBLE_DEVICES=0,2
megatron sft
--load Qwen3-30B-A3B-Instruct-2507-mcore
--dataset 'train_sft_samples_08_26_2025_ms_swift.json'
--packing true
--train_type lora
--lora_rank 64
--lora_alpha 64
--use_rslora true
--target_modules all-linear
--split_dataset_ratio 0.01
--expert_model_parallel_size 2
--moe_permute_fusion true
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_aux_loss_coeff 1e-3
--micro_batch_size 1
--global_batch_size 32
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--max_epochs 1
--finetune true
--cross_entropy_loss_fusion true
--loss_scale last_round
--lr 3e-5
--lr_warmup_fraction 0.05
--min_lr 1e-7
--save megatron_output/Qwen3-30B-A3B
--eval_interval 100
--save_interval 100
--max_length 12288
--num_workers 8
--dataset_num_proc 8
--no_save_optim true
--no_save_rng true
--sequence_parallel true
--attention_backend flash
--optimizer_cpu_offload true
--use_precision_aware_optimizer true
--optimizer_offload_fraction 1
--model_author swift
--model_name swift-robot