Segmentation fault on inference

[younes200]

Hi
Getting a 'Segmentation fault (core dumped)' error while trying to make an inference using the given command.

python inference/mimictalk_infer.py --drv_aud data/raw/examples/80_vs_60_10s.wav \ 
--drv_pose data/raw/examples/German_20s.mp4 \
--drv_style data/raw/examples/German_20s.mp4 \
--bg_img data/raw/examples/bg.png \
--torso_ckpt checkpoints_mimictalk/German_20s_22 \
--out_name output.mp4 --out_mode final

End output :

MimicTalk is rendering frames:   0%|                                                           | 0/748 [00:00<?, ?it/s]
Setting up PyTorch plugin "bias_act_plugin"... Done.
Segmentation fault (core dumped)

Anyone else having the same issue ?

(pytorch) root@77b88c1984d5:/workspace# python3 -m torch.utils.collect_env
/root/miniconda3/envs/pytorch/lib/python3.9/runpy.py:127: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.21 (main, Dec 11 2024, 16:24:11)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 550.127.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               30
On-line CPU(s) list:                  0-29
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7J13 64-Core Processor
CPU family:                           25
Model:                                1
Thread(s) per core:                   1
Core(s) per socket:                   1
Socket(s):                            30
Stepping:                             1
BogoMIPS:                             4899.99
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip pku ospke vaes vpclmulqdq rdpid fsrm arch_capabilities
Virtualization:                       AMD-V
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            1.9 MiB (30 instances)
L1i cache:                            1.9 MiB (30 instances)
L2 cache:                             15 MiB (30 instances)
L3 cache:                             480 MiB (30 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-29
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch3d==0.7.8
[pip3] torch==2.4.0+cu121
[pip3] torchaudio==2.4.0+cu121
[pip3] torchdiffeq==0.2.5
[pip3] torchode==1.0.0
[pip3] torchshow==0.5.1
[pip3] torchtyping==0.1.5
[pip3] torchvision==0.19.0+cu121
[pip3] triton==3.0.0
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] pytorch3d                 0.7.8                    pypi_0    pypi
[conda] torch                     2.4.0+cu121              pypi_0    pypi
[conda] torchaudio                2.4.0+cu121              pypi_0    pypi
[conda] torchdiffeq               0.2.5                    pypi_0    pypi
[conda] torchode                  1.0.0                    pypi_0    pypi
[conda] torchshow                 0.5.1                    pypi_0    pypi
[conda] torchtyping               0.1.5                    pypi_0    pypi
[conda] torchvision               0.19.0+cu121             pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi

[pete-j-turnbull]

I'm also getting this issue with the exact same setup as you - any idea how to debug this?

[younes200]

@pete-j-turnbull it's a problem with cuda version of bias_act and upfirdn2d, you can make it work to force using python version of this function by commenting this lines :

MimicTalk/modules/eg3ds/torch_utils/ops/bias_act.py

Line 86 in a79f90e

if impl == 'cuda' and x.device.type == 'cuda' and _init():

If you have better solution to use upfirdn2d_plugin and bias_act_plugin please let me know ;)

[pete-j-turnbull]

Very interesting thanks @younes200 !
It seems like using the python versions of those ops might cause a significant slow down - have you noticed inference taking a long time?

[younes200]

Very interesting thanks @younes200 ! It seems like using the python versions of those ops might cause a significant slow down - have you noticed inference taking a long time?

Yes, it slows down the inference. Also, during training, I had to force the image segmentation generation to run on a single process otherwise it stuck indefinably.

[pete-j-turnbull]

@younes200 If you'd like to chat on this I'm happy to put some time into getting the plugins fixed - might be helpful to get some more context that you may have though! :) Fire me an email at pete@yuzulabs.io if you're up for it.