训练报错

[yulj21]

CUDA_VISIBLE_DEVICES=1 python train_face.py --type face -s $dataset -m $workspace --init_num 2000 --densify_grad_threshold 0.0005 --audio_extractor $audio_extractor --pretrain_path $pretrain_face_path --iterations 50000 --sh_degree 1 --N_views $n_views
Optimizing ./workspace
Output folder: ./workspace [06/03 10:55:34]
Found transforms_train.json file, assuming Blender data set! [06/03 10:55:34]
Reading Training Transforms [06/03 10:55:34]
4500it [00:04, 1067.44it/s]
4500it [03:07, 24.03it/s]
Reading Test Transforms [06/03 10:58:46]
422it [00:00, 1063.45it/s]
422it [00:12, 33.05it/s]
Generating random point cloud (2000)... [06/03 10:58:59]
Loading Training Cameras [06/03 10:59:00]
Loading Test Cameras [06/03 10:59:06]
Number of points at initialisation : 2000 [06/03 10:59:06]
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] [06/03 10:59:13]
/home/appuser/yulj21/talking_face/instag/anaconda37/lib/python3.7/site-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
f"The parameter '{pretrained_param}' is deprecated since 0.13 and will be removed in 0.15, "
/home/appuser/yulj21/talking_face/instag/anaconda37/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /home/appuser/yulj21/talking_face/instag/anaconda37/lib/python3.7/site-packages/lpips/weights/v0.1/alex.pth [06/03 10:59:13]
Training progress: 0%| | 0/50000 [00:00<?, ?it/s]NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
19 [06/03 10:59:16]
tensor(16.0583, device='cuda:0', dtype=torch.float64) [06/03 10:59:16]
tensor(0.0645, device='cuda:0', dtype=torch.float64) [06/03 10:59:16]

[ITER 1] Evaluating test: L1 0.06452561836493642 PSNR 16.05832933124743 [06/03 10:59:16]
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
5 [06/03 10:59:18]

系统环境：
python 3.7.6
Package Version

---

absl-py 2.1.0
addict 2.4.0
alabaster 0.7.12
anaconda-client 1.7.2
anaconda-navigator 1.9.12
anaconda-project 0.8.3
argh 0.26.2
asn1crypto 1.3.0
astroid 2.3.3
astropy 4.0
atomicwrites 1.3.0
attrs 19.3.0
audioread 3.0.1
autopep8 1.4.4
Babel 2.8.0
backcall 0.1.0
backports.functools-lru-cache 1.6.1
backports.shutil-get-terminal-size 1.0.0
backports.tempfile 1.0
backports.weakref 1.0.post1
beautifulsoup4 4.8.2
bitarray 1.2.1
bkcharts 0.2
bleach 3.1.0
bokeh 1.4.0
boto 2.49.0
Bottleneck 1.3.2
cachetools 5.5.2
cargo 0.3
certifi 2019.11.28
cffi 1.14.0
chardet 3.0.4
Click 7.0
cloudpickle 1.3.0
clyent 1.2.2
colorama 0.4.3
conda 4.8.2
conda-build 3.18.11
conda-package-handling 1.6.0
conda-verify 3.4.2
ConfigArgParse 1.7
contextlib2 0.6.0.post1
cryptography 2.8
cycler 0.10.0
Cython 0.29.15
cytoolz 0.10.1
dask 2.11.0
decorator 4.4.1
defusedxml 0.6.0
diff-gauss 1.0.10.0
diff-match-patch 20181111
distributed 2.11.0
docutils 0.16
entrypoints 0.3
et-xmlfile 1.0.1
face-alignment 1.4.1
fastcache 1.1.0
filelock 3.0.12
flake8 3.7.9
Flask 1.1.1
fsspec 0.6.2
future 0.18.2
gevent 1.4.0
glob2 0.7
gmpy2 2.0.8
google-auth 2.38.0
google-auth-oauthlib 0.4.6
greenlet 0.4.15
gridencoder 0.0.0
grpcio 1.62.3
h5py 2.10.0
HeapDict 1.0.1
html5lib 1.0.1
huggingface-hub 0.16.4
hypothesis 5.5.4
idna 2.8
imageio 2.6.1
imageio-ffmpeg 0.5.1
imagesize 1.2.0
importlib-metadata 6.7.0
importlib-resources 5.12.0
intervaltree 3.0.2
ipykernel 5.1.4
ipython 7.12.0
ipython-genutils 0.2.0
ipywidgets 7.5.1
isort 4.3.21
itsdangerous 1.1.0
jaraco.classes 3.2.3
jdcal 1.4.1
jedi 0.14.1
jeepney 0.4.2
Jinja2 2.11.1
joblib 0.14.1
json5 0.9.1
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 5.3.4
jupyter-console 6.1.0
jupyter-core 4.6.1
jupyterlab 1.2.6
jupyterlab-server 1.0.6
keyring 21.1.0
keyrings.alt 4.2.0
kiwisolver 1.1.0
lazy-loader 0.4
lazy-object-proxy 1.4.3
libarchive-c 2.8
librosa 0.10.2.post1
lief 0.9.0
llvmlite 0.31.0
locket 0.2.0
lpips 0.1.4
lxml 4.5.0
Markdown 3.4.4
markdown-it-py 2.2.0
MarkupSafe 2.1.5
matplotlib 3.1.3
mccabe 0.6.1
mdurl 0.1.2
mistune 0.8.4
mkl-fft 1.0.15
mkl-random 1.1.0
mkl-service 2.3.0
mmcv-full 1.7.1
mock 4.0.1
model-index 0.1.11
more-itertools 8.2.0
mpmath 1.1.0
msgpack 1.0.5
multipledispatch 0.6.0
navigator-updater 0.2.1
nbconvert 5.6.1
nbformat 5.0.4
networkx 2.4
nltk 3.4.5
nose 1.3.7
notebook 6.0.3
numba 0.48.0
numexpr 2.7.1
numpy 1.21.6
numpydoc 0.9.2
oauthlib 3.2.2
olefile 0.46
opencv-python 4.11.0.86
openmim 0.3.8
openpyxl 3.0.3
ordered-set 4.1.0
packaging 20.1
pandas 1.0.1
pandocfilters 1.4.2
parso 0.5.2
partd 1.1.0
path 13.1.0
pathlib2 2.3.5
pathtools 0.1.2
patsy 0.5.1
pep8 1.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.5.0
pip 20.0.2
pkginfo 1.5.0.1
platformdirs 4.0.0
pluggy 0.13.1
ply 3.11
plyfile 0.8.1
pooch 1.8.2
prettytable 3.7.0
prometheus-client 0.7.1
prompt-toolkit 3.0.3
protobuf 3.20.3
psutil 5.6.7
ptyprocess 0.6.0
py 1.8.1
pyasn1 0.5.1
pyasn1-modules 0.3.0
PyAudio 0.2.14
pycodestyle 2.5.0
pycosat 0.6.3
pycparser 2.19
pycrypto 2.6.1
pycurl 7.43.0.5
pydocstyle 4.0.1
pyflakes 2.1.1
pygments 2.17.2
pylint 2.4.4
pyodbc 4.0.0-unsupported
pyOpenSSL 19.1.0
pyparsing 2.4.6
pyrsistent 0.15.7
PySocks 1.7.1
pytest 5.3.5
pytest-arraydiff 0.3
pytest-astropy 0.8.0
pytest-astropy-header 0.1.2
pytest-doctestplus 0.5.0
pytest-openfiles 0.4.0
pytest-remotedata 0.3.2
python-dateutil 2.8.1
python-jsonrpc-server 0.3.4
python-language-server 0.31.7
python-speech-features 0.6
pytz 2019.3
PyWavelets 1.1.1
pyxdg 0.26
PyYAML 5.3
pyzmq 18.1.1
QDarkStyle 2.8
QtAwesome 0.6.1
qtconsole 4.6.0
QtPy 1.9.0
regex 2024.4.16
requests 2.22.0
requests-oauthlib 2.0.0
resampy 0.4.3
rich 13.8.1
rope 0.16.0
rsa 4.9
Rtree 0.9.3
ruamel-yaml 0.15.87
scikit-image 0.16.2
scikit-learn 0.22.1
scipy 1.4.1
seaborn 0.10.0
SecretStorage 3.1.2
Send2Trash 1.5.0
setuptools 45.2.0.post20200210
simple-knn 0.0.0
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.14.0
snowballstemmer 2.0.0
sortedcollections 1.1.2
sortedcontainers 2.1.0
soundfile 0.13.1
soupsieve 1.9.5
soxr 0.3.7
Sphinx 2.4.0
sphinxcontrib-applehelp 1.0.1
sphinxcontrib-devhelp 1.0.1
sphinxcontrib-htmlhelp 1.0.2
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.2
sphinxcontrib-serializinghtml 1.1.3
sphinxcontrib-websupport 1.2.0
spyder 4.0.1
spyder-kernels 1.8.1
SQLAlchemy 1.3.13
statsmodels 0.11.0
sympy 1.5.1
tables 3.6.1
tabulate 0.9.0
tblib 1.6.0
tensorboard 2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorboardX 2.6.2.2
terminado 0.8.3
testpath 0.4.4
tokenizers 0.12.1
toolz 0.10.0
torch 1.12.1+cu113
torch-ema 0.3
torchaudio 0.12.1+cu113
torchvision 0.13.1+cu113
tornado 6.0.3
tqdm 4.42.1
traitlets 4.3.3
transformers 4.20.0
typing-extensions 4.7.1
ujson 1.35
unicodecsv 0.14.1
urllib3 1.25.8
watchdog 0.10.2
wcwidth 0.1.8
webencodings 0.5.1
Werkzeug 2.2.3
wheel 0.34.2
widgetsnbextension 3.5.1
wrapt 1.11.2
wurlitzer 2.0.0
xlrd 1.2.0
XlsxWriter 1.2.7
xlwt 1.3.0
xmltodict 0.12.0
yapf 0.28.0
zict 1.0.0
zipp 3.15.0

[Fictionarry]

看起来是环境问题，但在两台机器上均未见过相似的情况，尝试使用推荐的环境重装看看，并检查一下全局cuda版本与虚拟环境中pytorch的cuda版本是否一致

[yulj21]

我这边查看结果是，cuda环境跟您给的是一样的 我这边把
#if not mode_long and iteration > warm_step + 2000:
if 0:
loss += 0.01 * (1 - viewpoint_cam.talking_dict["normal"].cuda() * render_pkg["normal"]).sum(0)[head_mask^mouth_mask].mean()
if iteration % opt.opacity_reset_interval > 100:
# depth_normal = depth_to_normal(viewpoint_cam, render_pkg["depth"]).permute(2,0,1)
# loss += 0.001 * (1 - viewpoint_cam.talking_dict["normal"].cuda() * depth_normal).sum(0)[face_mask^mouth_mask].mean()

                depth = render_pkg["depth"][0]
                depth_mono = viewpoint_cam.talking_dict['depth'].cuda()
                loss_1 = 1e-2 * (normalize(depth)[face_mask^mouth_mask] - normalize(depth_mono)[face_mask^mouth_mask]).abs().mean()
                print(loss_1)
                loss += loss_1

这段代码注释掉后能正常跑

[Fictionarry]

我这边查看结果是，cuda环境跟您给的是一样的 我这边把 #if not mode_long and iteration > warm_step + 2000: if 0: loss += 0.01 * (1 - viewpoint_cam.talking_dict["normal"].cuda() * render_pkg["normal"]).sum(0)[head_mask^mouth_mask].mean() if iteration % opt.opacity_reset_interval > 100: # depth_normal = depth_to_normal(viewpoint_cam, render_pkg["depth"]).permute(2,0,1) # loss += 0.001 * (1 - viewpoint_cam.talking_dict["normal"].cuda() * depth_normal).sum(0)[face_mask^mouth_mask].mean()

                depth = render_pkg["depth"][0]
                depth_mono = viewpoint_cam.talking_dict['depth'].cuda()
                loss_1 = 1e-2 * (normalize(depth)[face_mask^mouth_mask] - normalize(depth_mono)[face_mask^mouth_mask]).abs().mean()
                print(loss_1)
                loss += loss_1

这段代码注释掉后能正常跑

可以尝试一下使用相同环境和未修改过的原始仓库代码跑 https://github.com/sstzal/DFRF/tree/main/dataset/vids 中的cnn视频。如果还是报nan的话应该是rasterizer没有正常安装，即使能跑起来可能还是会有问题。如果能成功跑且结果正常的话环境应该就没什么问题

[yulj21]

我这边查看结果是，cuda环境跟您给的是一样的 我这边把 #if not mode_long and iteration > warm_step + 2000: if 0: loss += 0.01 * (1 - viewpoint_cam.talking_dict["normal"].cuda() * render_pkg["normal"]).sum(0)[head_mask^mouth_mask].mean() if iteration % opt.opacity_reset_interval > 100: # depth_normal = depth_to_normal(viewpoint_cam, render_pkg["depth"]).permute(2,0,1) # loss += 0.001 * (1 - viewpoint_cam.talking_dict["normal"].cuda() * depth_normal).sum(0)[face_mask^mouth_mask].mean()

                depth = render_pkg["depth"][0]
                depth_mono = viewpoint_cam.talking_dict['depth'].cuda()
                loss_1 = 1e-2 * (normalize(depth)[face_mask^mouth_mask] - normalize(depth_mono)[face_mask^mouth_mask]).abs().mean()
                print(loss_1)
                loss += loss_1

这段代码注释掉后能正常跑

可以尝试一下使用相同环境和未修改过的原始仓库代码跑 https://github.com/sstzal/DFRF/tree/main/dataset/vids 中的cnn视频。如果还是报nan的话应该是rasterizer没有正常安装，即使能跑起来可能还是会有问题。如果能成功跑且结果正常的话环境应该就没什么问题

好 我先跑下看下，结果出来后在回复您

[yulj21]

我跑了下示例中的视频没有问题，但是我用我的好几个视频都不通过，
我突然想起来，我之前用talkingface这个我参考了下geneface的相机参数，能正常跑了