RePaint

Inpainting using Denoising Diffusion Probabilistic Models [Paper]

Code will be released soon

Star this project to get notified.

RePaint fills a missing image part using diffusion models

What are the blue parts?
Those parts are missing and therefore have to be filled by RePaint.
RePaint generates the missing parts inspired by the known parts.

How does it work?
RePaint starts from pure noise. Then the image is denoised step-by-step.
It uses the known part to fill the unknown part in each step.

Why does the noise level fluctuate during generation?
Our noise schedule improves the harmony between the generated and
the known part [4.2 Resampling].

RePaint on diverse content and shapes of missing regions

The blue region is unknown and filled by RePaint:

Note: RePaint creates many meaningful fillings.

1. Face: Expressions and features like an earring or a mole.
   
2. Computer: The computer screen shows different images, text, and even a logo.
   
3. Greens: RePaint makes sense of the tiny known part and incorporates it in a beetle, spaghetti, and plants.
   
4. Garden: From simple filling like a curtain to complex filling like a human.
   

Extreme Case 1: Generate every second line

• Every Second line of the input image is unknown.
• Most inpainting methods fail on such masks.

Extreme Case 2: Upscale an image

• The inpainting only knows pixels with a strided access of 2.
• A ratio of 3/4 of the image has to be filled.
• This is equivalent to Super-Resolution with Nearest Neighbor kernel.

RePaint conditions the diffusion model on the known part

• RePaint uses Denoising Diffusion Probabilistic Models.
• We condition this process on the given image content.

Intuition of one denoising step:

1. Sample the known part: Add gaussian noise to the known regions of the image.
   We obtain a noisy image that follows the denoising process exactly.
2. Denoise one step: Denoise the previous image for one step. This generates
   content for the unknown region conditioned on the known region.
3. Join: Merge the images from both steps.

Details are in Algorithm 1 on Page 4. [Paper]

How to harmonize the generated with the known part?

• Fail: When using only the algorithm above, the filling is not well harmonized with the known part (n=1).
• Fix: When applying the [4.2 Resampling] technique, the images are better harmonized (n>1).

RePaint Fails

• The ImageNet model is biased towards inpainting dogs.
• This is due to the high ratio of dog images in ImageNet.

User Study State-of-the-Art Comparison

• Outperforms autoregressive-based and GAN-based SOTA methods for
  all masks with significance 95% except for two inconclusive cases.
• The user study was done for six different masks on three datasets.
• RePaint outperformed SOTA methods in 42 of 44 cases. [Paper]

Explore the Visual Examples

• Datasets: CelebA-HQ, ImageNet, Places2
• Masks: Random strokes, half image, huge, sparse
• Explore more examples like this in the [Appendix].

Code will be released soon

Star this project to get notified.