lcm lora can't load

[hanggun]

Dear author, thank you for your valuable work. I have encounter the problem of loading LCM lora. Can you see why the Lcm lora can not be loaded into the RORem mixed model?
I used following code.

pipe_edit = AutoPipelineForInpainting.from_pretrained(
        '/home/workspace/volume3/ps/pretrained_model/sd/stable-diffusion-xl-1.0-inpainting-0.1/',
        torch_dtype=torch.float16, 
        variant="fp16"
    )
unet = UNet2DConditionModel.from_pretrained('/home/workspace/volume3/ps/pretrained_model/RORem/').to("cuda",dtype=torch.float16)
pipe_edit.unet = unet
pipe_edit.scheduler = LCMScheduler.from_config(pipe_edit.scheduler.config)
pipe_edit.load_lora_weights("/home/workspace/volume3/ps/pretrained_model/RORem_lcm/pytorch_lora_weights.safetensors")
pipe_edit.to('cuda')

But encounter

Loading adapter weights from state_dict led to missing keys in the model: down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_q.lora_A.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_q.lora_B.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_k.lora_A.default_0.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_k.lora_B.default_0.weight

[leeruibin]

Which diffusers version do you use? Sometimes, the lastest diffusers may have different behavior compared with previous version.

[hanggun]

Which diffusers version do you use? Sometimes, the lastest diffusers may have different behavior compared with previous version.

I tried 0.30.0 which is the same as yours and the latest version, all can not

[leeruibin]

well, this is strange, seems like when you load_lora_weights, the expected key become

down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_q.lora_A.default_0.weight

instead of the key in the
unet.down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_q.lora_A.default_0.weight
keys in the pretrained LCM are starting with "unet"

I guess this the reason why you fail to load the LoRA,

    import safetensors
    import safetensors.torch  

    def inspect_safetensors_keys(file_path):
        """
        check keys in the safetensors

        Args:
            file_path (str): safetensors file path
        """
        try:
            metadata = safetensors.safe_open(file_path, framework="pt")
            
            if metadata is not None:
                print("Keys in the safetensors file:")
                for key in metadata.keys():
                    print(f"- {key}")
            else:
                print("No metadata found in the safetensors file.")
        
        except Exception as e:
            print(f"Error reading safetensors file: {e}")

    file_path = "xxx"  # LCM_LoRA path
    inspect_safetensors_keys(file_path)

you can use this code the check the keys in the "pytorch_lora_weights.safetensors". Currently I have no idea which operator do you use to cause this shift.

Currently, the loading code works for me, a possible solution for you is that you change the key in the LoRA, for example, delete the "unet." in your case.