-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Config for the mic scene #29
Comments
The defaults should work, what problems are you running into? |
The defaults work for most of the other blender sequences, but in case of (I'll try a couple other random seeds and keep you posted) |
We did find that for the blender scenes with a lot of whitespace (Ficus and Mic) the training would sometimes diverge right at the beginning of training. One hack that helps with initial training stability for these scenes with a lot of background and a small object of interest is to crop out the central part of the image for training for just the first 1000 iters or so, so it's not being supervised on almost all white pixels at the start. |
Ah thanks for the note, I briefly pondered over cropping, but this is reassuring! |
For future use, just added this functionality: fd624c0 Something like |
Another question on the white background: Is your PSNR score (and others) calculated with the whole image? Actually the white background is quite large (about 60% to 70% of the pixels), using pretrained model gives 40 to 50 PSNR on white regions and ~30 on object region. Also, won't training on large fraction on white background makes the model optimize more for white background? i.e. it predicts perfectly the white part but slightly worse on object. That's not what we want right? How about training on object region only? |
@kwea123 While training on "object region only" is a fair thing to do for synthetic objects (like the Blender dataset), it's seldom applicable to real scenes. Eg. if you look at LLFF data, you'll need to reason over the entire image and it's hard to ignore pixels, unless you bake in additional prior knowledge about the scene. This is my perspective though; I'd be keen to hear if others feel the same way too. |
@kwea123 We calculate all metrics with the whole image. It is true that training more on the object region than the background improves overall metrics -- however, you still need to send some amount of the samples through the background pixels (10-20%) or else you will end up with floater artifacts when you render whole images at test time. In general I tend to agree with @krrish94 that it's best to avoid using object masks with image based supervision since it limits the applicability to real scenes. |
A late reply of what I just found recently! (For my pytorch implementation) Sometimes the network initialization makes the sigma output always zero after the relu activation (i.e. the ray's color is predicted white), and since there is a lot of white space, the training will not escape from this local minimum, making the result all white. Therefore, I tried replacing |
The
|
Thank you for your wonderful work about re-implement the NERF-IN-Wild, However, from the original nerf-wild paper and the new SIGGRAPH paper 《ReLU Fields: The Little Non-linearity That Could》 suggests, it seems the relu function is important in the nerf training. I think maybe there are some other mistake cause this problem. |
I too am trying to repro vanilla NeRF and finding it hard on mic & ficus. Could you please confirm whether in the original NeRF paper you used this trick? Or did you get it to eventually train without cropping, just after trying enough times? |
Hi @bmild,
Do you happen to have a separate config for the
mic
scene? (or the defaults for blender sequences are expected to work?)The text was updated successfully, but these errors were encountered: