Wrong input for generating self.D in models/cgan_model.py line 44

[Agchai52]

Thank you for your excellent work! I had been searching for tf version of DeblurGAN for months.

I find a bug in "models/cgan_model.py" Line 44.
self.D = discriminator(tf.concat([self.G, self.input['real_img']], axis=0))
which means that the input of discriminator is "[deblurred image from G, sharp image]", where real_img = sharp image.

It will effect Line 100, computing the adversarial loss,
self.adv_loss = adv_loss(self.D)

However in Kupyn's Pytorch code, "models/conditional_gan_model.py" Line 95, the corresponding code of Line 44 and Line 100 is
self.loss_G_GAN = self.discLoss.get_g_loss(self.netD, self.real_A, self.fake_B).
which means the input of discriminator is "[deblurred image from G, blurry image]", since self.real_A = blurry image and the self.fake_B = deblurred image from G.
That is is how Kupyn generates the adversarial loss.

So, the correct Line 44 should be:

self.D = discriminator(tf.concat([self.G, self.input['blur_img']], axis=0))

From the point of view of Conditional GAN, the blurry image is the auxiliary information for both generator and discriminator. In other words, the blurry image is the information we condition on.
the input of the discriminator should be [G(blurry), blurry] or [real, blurry].

[LeeDoYup]

Thank you @Agchai52 for you careful code review review.
https://github.com/KupynOrest/DeblurGAN/blob/39e8a4b408b90d0ef98608c5c4562eae3e184251/models/losses.py#L157-L167

I checked the codes which you refer.
In the case of discriminator, the input is [generated image, real image].

In Conditional GAN, the blurry image is the auxiliary information for (not both) generator only.
After condition of blur image, generator creates fake image.
Then, discriminator try to classify the fake image and the real (sharp) image.

However, in the review process, i found another bug for generator loss function.
18a7f36

Thank you.

[Agchai52]

Thank you for your response!

Both Discriminator and Generator do condition on auxiliary information (blurry).

where x is auxiliary imformation or the observed image (blurry image), y is the sharp image (real image) and z is the random noise. Both in the first term and second term, the inputs of D contain x (blurry image).

Eq. (1) comes from paper "Image-to-Image Translation with Conditional Adversarial Networks" by Isola et al. This paper is one of the most important references of DeblurGAN. We can also check Eq. (2) in the original conditional GAN paper "Conditional Generative Adversarial Nets" by Mirza et al. Although notation is different, the same conclusion can be drawn.

[LeeDoYup]

I understand the equation and the condition of blur image is important also in discriminator.
However, first, the discriminator loss need two result image: real_img and generated_img.

How can we calculate D(x,y) ??
Does it means ??

G_input = tf.concat([self.G, self.input['blur_img'], axis=0)
D_input = tf.concat([self.input['real_img'], self.input['blur_img']], axis=0)
self.D = discriminator(tf.concat([G_input, D_input] , axis=0)

Meanwhile, the reason why i concat generated_image and real image is to output two probability (first_term and second term in (1))

[LeeDoYup]

However I found that the your reference DeblurGAN code (pytorch, original author) do not concat the blur image into the input of discriminator

[Agchai52]

That a very quick response!

I just checked the link your shared. The problem is actually on DeblurGAN. You just follow their code. I didn't dive into their function. Sorry about that. (the pytorch DeblurGAN is too painful to read, forgive me...)

The way to use auxiliary info is exactly what you show. Just concatinate the blurry after the real and the generated. To get D(x,y) and D(x, G(x,z)), we need to feed them into Discrimiantor separately, rather than together.

You can check this link for more details: https://github.com/yenchenlin/pix2pix-tensorflow/blob/master/model.py
Line 75-83.

This the tf version of Image-to-image.

[LeeDoYup]

Thank you. I didn't check the original pix2pix codes carefully, but your comments are really helpful to me.

Your comment is right, most of GAN models are implement to feed generator and discriminator separately.
However, after i read the DeblurGAN paper, the model doesn't need to do.
So i concat the self.G and self.input['real_img'] and feed together.
(The model use batch_size=1. Memory is enough but separate feed makes the training slow.
Of course, to use gpu fully, i will additionally implement the data_feeder module with tf.data)

[Agchai52]

So, DeblurGAN doesn't use conditional GAN as the claimed.
That makes sense!
Thank you for your reply!
Great code!