关于运行的问题

[HenryFordham]

作者您好，我按照您的代码运行了一下，下载了各种大模型和您的模型

但是运行起来有如下错误。我不知道是怎么回事，看上去是模型的encode部分出现了哪些错误。如果您知道，请指教，谢谢

sampling 50 steps using ddpm sampler
Traceback (most recent call last):
File "inference_partition.py", line 160, in
main()
File "inference_partition.py", line 141, in main
preds, bpp = process(
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "inference_partition.py", line 56, in process
"c_crossattn": [model.get_learned_conditioning([""] * n_samples)]
File "/root/autodl-tmp/DiffEIC/ldm/models/diffusion/ddpm.py", line 677, in get_learned_conditioning
c = self.cond_stage_model.encode(c)
File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 236, in encode
return self(text)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 213, in forward
z = self.encode_with_transformer(tokens.to(next(self.model.parameters()).device))
File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 220, in encode_with_transformer
x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask)
File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 232, in text_transformer_forward
x = r(x, attn_mask=attn_mask)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/open_clip/transformer.py", line 263, in forward
x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask))
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/open_clip/transformer.py", line 250, in attention
return self.attn(
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1275, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/functional.py", line 5438, in multi_head_attention_forward
raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1).

[HenryFordham]

我是运行了这个命令 ：python3 inference_partition.py --ckpt_sd ./weights/v2-1_512-ema-pruned.ckpt --ckpt_lc ./weights/lc.ckpt --config configs/model/diffeic.yaml --input /root/autodl-tmp/kodak/ --output ./compress_image/ --steps 50 --device cuda

[huai-chang]

File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 213, in forward
z = self.encode_with_transformer(tokens.to(next(self.model.parameters()).device))
File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 220, in encode_with_transformer
x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask)
File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 232, in text_transformer_forward
x = r(x, attn_mask=attn_mask)

从这里来看，是使用CLIP提取文本特征时出错了，可能是你的库的版本不对？我试了一下，是能够正常运行的，且attn_mask的大小也是torch.Size([77, 77])，和你这里是一样的

[HenryFordham]

好啦！运行起来了，还有个问题想问您。就是我看您先用了个ELIC的编码器，同时用了一个autoencoder作编码，这个autoencoder是固定参数的，那ELIC的参数是和unet同时参与训练的吗？如果是的话为啥CDDM模块中那个denoiser是画了个锁的呢？

[huai-chang]

这里使用的是ControlNet中提出的策略。为了防止破坏stable diffusion的生成能力，会把噪声估计器固定，然后使用一个额外的可训练控制模块(Control Module)注入外部条件信息。画锁表示这个网络的参数在训练过程中不进行更新。

[HenryFordham]

我还有最后一个小问题🤣，就是我看到你论文和损失函数里面写的是BPP+diffusion生成的loss，以及一个约束autoencoder编码器和zc的loss。但是按照end to end编码器来说，损失都设定为bpp+MSE（x，x^)，如果没有MSE（x，x^)，怎么保证生成的图片和原图是尽可能相似的呢？🤣

[huai-chang]

首先，Stable Diffusion 的扩散操作是在潜在扩散空间中进行。所以只需要生成的潜在表示 $$\hat{z_0}$$ 和原始图像的潜在表示 $$z_0$$ 能够一致就行。使用噪声估计损失是能够做到这点的，只要每一步都能相对准确地将噪声去除。具体可以看看扩散模型相关的论文。

其次，由于在DiffEIC中需要50个采样步骤来重建图像，所以直接使用 $$MSE(x, \hat{x})$$ 进行优化需要巨大的显存，这是不现实的。一种可行的方式是减少重建所需的去噪步数，以便能够在计算资源有限的情况下直接使用 $$MSE(x, \hat{x})$$ 作为优化损失。这个可以看看我们的另一个工作RDEIC🫣，我们在第二阶段使用了 $$MSE(x, \hat{x})$$ 作为损失。

[HenryFordham]

谢谢啦～

[paomian001]

@HenryFordham 您好，打扰了，我也遇到了同样的问题，请问您方便告知您是如何解决该问题的？

[paomian001]

作者您好，我按照您的代码运行了一下，下载了各种大模型和您的模型

但是运行起来有如下错误。我不知道是怎么回事，看上去是模型的encode部分出现了哪些错误。如果您知道，请指教，谢谢

sampling 50 steps using ddpm sampler Traceback (most recent call last): File "inference_partition.py", line 160, in main() File "inference_partition.py", line 141, in main preds, bpp = process( File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "inference_partition.py", line 56, in process "c_crossattn": [model.get_learned_conditioning([""] * n_samples)] File "/root/autodl-tmp/DiffEIC/ldm/models/diffusion/ddpm.py", line 677, in get_learned_conditioning c = self.cond_stage_model.encode(c) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 236, in encode return self(text) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 213, in forward z = self.encode_with_transformer(tokens.to(next(self.model.parameters()).device)) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 220, in encode_with_transformer x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 232, in text_transformer_forward x = r(x, attn_mask=attn_mask) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/open_clip/transformer.py", line 263, in forward x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask)) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/open_clip/transformer.py", line 250, in attention return self.attn( File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1275, in forward attn_output, attn_output_weights = F.multi_head_attention_forward( File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/functional.py", line 5438, in multi_head_attention_forward raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.") RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1).

您好 我也遇到了同样的问题，请问您是如何解决的

[huai-chang]

作者您好，我按照您的代码运行了一下，下载了各种大模型和您的模型
但是运行起来有如下错误。我不知道是怎么回事，看上去是模型的encode部分出现了哪些错误。如果您知道，请指教，谢谢
sampling 50 steps using ddpm sampler Traceback (most recent call last): File "inference_partition.py", line 160, in main() File "inference_partition.py", line 141, in main preds, bpp = process( File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "inference_partition.py", line 56, in process "c_crossattn": [model.get_learned_conditioning([""] * n_samples)] File "/root/autodl-tmp/DiffEIC/ldm/models/diffusion/ddpm.py", line 677, in get_learned_conditioning c = self.cond_stage_model.encode(c) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 236, in encode return self(text) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 213, in forward z = self.encode_with_transformer(tokens.to(next(self.model.parameters()).device)) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 220, in encode_with_transformer x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask) File "/root/autodl-tmp/DiffEIC/ldm/modules/encoders/modules.py", line 232, in text_transformer_forward x = r(x, attn_mask=attn_mask) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/open_clip/transformer.py", line 263, in forward x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask)) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/open_clip/transformer.py", line 250, in attention return self.attn( File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1275, in forward attn_output, attn_output_weights = F.multi_head_attention_forward( File "/root/miniconda3/envs/mfr/lib/python3.8/site-packages/torch/nn/functional.py", line 5438, in multi_head_attention_forward raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.") RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1).

您好 我也遇到了同样的问题，请问您是如何解决的

open_clip_torch==2.22.0
openai_clip==1.0.1
可以检查一下这两个库的版本是否一致