Support PuLID

[huchenlei]

Closes #2835.

Overview

PuLID is an ip-adapter alike method to restore facial identity. It uses both insightface embedding and CLIP embedding similar to what ip-adapter faceid plus model does. However, there is an extra process of masking out the face from background environment using facexlib before passing image to CLIP. PuLID also uses Eva CLIP instead of normal CLIP. In the attn overrides, PuLID also does something more than IPAdapter, as it zero pads the tensor and adds to ortho of hidden states. If you are interested you can read their paper on how this is handled.

How to use

Install sd-webui-controlnet-evaclip extension

PuLID uses both evaclip embedding and insightface embedding as proj module input. So in order to use the feature, you need to install https://github.com/huchenlei/sd-webui-controlnet-evaclip extension.

Download Model

You can download fp16 model here: https://huggingface.co/huchenlei/ipadapter_pulid/resolve/main/ip-adapter_pulid_sdxl_fp16.safetensors

ControlNet Unit Setting

There is an extra radio group to let you select PuLID mode. According to my testing, they only make slight difference, but still something nice to have.

Example

Input

Output [Fidelity]:

Output [Style]:

Generation params

masterpiece painting, buildings in the backdrop, kaleidoscope, lilac orange blue cream fuchsia bright vivid gradient colors, the scene is cinematic, (1girl:1.2), portrait,, emotional realism, double exposure, watercolor ink pencil, graded wash, color layering, magic realism, figurative painting, intricate motifs, organic tracery, pol
Negative prompt: flaws in the eyes, flaws in the face, flaws, lowres, non-HDRi, low quality, worst quality,artifacts noise, text, watermark, glitch, deformed, mutated, ugly, disfigured, hands, low resolution, partially rendered objects, deformed or partially rendered eyes, deformed, deformed eyeballs, cross-eyed,blurry
Steps: 20, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 3390099943, Size: 1024x1024, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, ControlNet 0: "Module: ip-adapter-auto, Model: ip-adapter_pulid_sdxl_fp16 [d86d05ea], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Version: v1.9.3-2-g3d3fc81f

Note

Comparing to instant id, you need stronger prompt to make the output stylized. Following is the same neon style generated by instant id:

Known issue

facexlib and evaclip in total consumes about 2~3GB of vram. So if you have low vram, that can be an issue. There is also about 500MB of vram not unloaded after unloading the preprocessors. I will need to investigate more on why vram here is not properly freed.

[George0726]

Hi author, thanks for the amazing support! I am trying to follow your PuLID mode and padding/ orthogonal stuff in code, however, the author seems not illustrating this part in the paper?

[yanghongquan521]

EVA02_CLIP_L_336_psz14_s6B 放在哪个文件夹下面

[maqianzhao]

EVA02_CLIP_L_336_psz14_s6B 哪里下载？

[dan4ik94]

@huchenlei Forge support please

[prog-ape]

@huchenlei Forge support please

[eurotaku]

@huchenlei
any idea why the previews first show a great representation of the source image and then it falls apart so awfully at the end? played around a bit with the weights and especially block 5 and 6 seem to be responsible for that overburned end result. but they also seem to have the main effect on the reproduction of the person's likeness.

https://github.com/Mikubill/sd-webui-controlnet/assets/129174287/d5572282-16f7-4df5-ac4d-ef82bd5e27bb