Exception in save_pretrained due to recent changes

guygurari
Sun, 23 Jun 2024 20:51:02 +0000

The error I’m getting:

Traceback (most recent call last):
  File "quantize_fp8.py", line 39, in <module>
    model.save_quantized(quantized_model_dir)
  File "/root/AutoFP8/auto_fp8/modeling.py", line 133, in save_quantized
    save_quantized_model(
  File "/root/AutoFP8/auto_fp8/quantize.py", line 342, in save_quantized_model
    model.save_pretrained(save_dir)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2661, in save_pretrained
    shard = {tensor: state_dict[tensor] for tensor in tensors}
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2661, in <dictcomp>
    shard = {tensor: state_dict[tensor] for tensor in tensors}
NameError: free variable 'state_dict' referenced before assignment in enclosing scope

This seems to be due to 2 conflicting changes from the last two weeks, which cause the loop over shards in save_pretrained to look like this:

        # Save the model
        for shard_file, tensors in state_dict_split.filename_to_tensors.items():
            shard = {tensor: state_dict[tensor] for tensor in tensors}   # state_dict is used here, but is deleted below
            # remake shard with onloaded parameters if necessary
            if module_map:
                ...
                state_dict = {name: "" for name in shard}   # overwrites state_dict
                ...
                del state_dict  # deletes state_dict

Screenshot of where the conflicting changes came from is attached.

Read full topic