Skip to content

Issues in Inference #12

New issue
New issue
Open
Open
Issues in Inference#12
@ghafoorharis

Description

@ghafoorharis
ghafoorharis
opened on Oct 21, 2025

Hi I followed the instructions and got the following error
command: export PYTHONPATH=$PYTHONPATH:. && mim test mmdet configs/Ul traSAM/UltraSAM_zero_shot/UltraSAM_box_refine_BUSI.py --checkpoint checkpoint/UltraSam.pth
Output:
System environment:
sys.platform: linux
Python: 3.10.18 (main, Jun 5 2025, 13:14:17) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 435233536
GPU 0,1: NVIDIA RTX A6000
CUDA_HOME: /usr/local/cuda-12.8
NVCC: Cuda compilation tools, release 12.8, V12.8.61
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
PyTorch: 2.9.0+cu128
PyTorch compiling details: PyTorch built with:

  • GCC 13.3

  • C++ Version: 201703

  • Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications

  • Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)

  • OpenMP 201511 (a.k.a. OpenMP 4.5)

  • LAPACK is enabled (usually provided by MKL)

  • NNPACK is enabled

  • CPU capability usage: AVX2

  • CUDA Runtime 12.8

  • NVCC architecture flags: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_100,code=sm_100;-gencode;arch=compute_120,code=sm_120

  • CuDNN 91.0.2 (built against CUDA 12.9)

    • Built with CuDNN 90.8

  • Magma 2.6.1

  • Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0fabc3ba44823f257e70ce397d989c8de5e362c1, CUDA_VERSION=12.8, CUDNN_VERSION=9.8.0, CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,

    TorchVision: 0.24.0+cu128
    OpenCV: 4.12.0
    MMEngine: 0.10.7

Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: 435233536
Distributed launcher: none
Distributed training: False
GPU number: 1

10/21 15:21:58 - mmengine - INFO - Config:
N_POINTS = 1
backend_args = None
custom_hooks = [
dict(type='MonkeyPatchHook'),
]
custom_imports = dict(
allow_failed_imports=False,
imports=[
'mmpretrain.models',
'endosam.datasets.transforms.custom_pipeline',
'endosam.datasets.transforms.point_formatting',
'endosam.visualization.point_visualization',
'endosam.models.detectors.SAM',
'endosam.models.backbones.vit_sam',
'endosam.models.backbones.MED_SA',
'endosam.models.dense_heads.sam_mask_decoder',
'endosam.models.dense_heads.sam_mask_class_decoder',
'endosam.datasets.evaluation.LabelMetric',
'endosam.models.utils.sam_layers',
'endosam.models.task_modules.assigners.SAMassigner',
'endosam.models.task_modules.prior_generators.prompt_encoder',
'endosam.models.task_modules.prior_generators.label_encoder',
'endosam.hooks.MonkeyPatchHook',
'endosam.hooks.FreezeHook',
'endosam.hooks.ValLossHook',
])
data_preprocessor = dict(
bgr_to_rgb=True,
mean=[
123.675,
116.28,
103.53,
],
pad_size_divisor=1024,
std=[
58.395,
57.12,
57.375,
],
type='DetDataPreprocessor')
data_root = 'datasets'
dataset_type = 'CocoDataset'
default_hooks = dict(
checkpoint=dict(
by_epoch=False,
interval=2000,
max_keep_ckpts=1,
save_best='coco/segm_mAP',
save_optimizer=False,
type='CheckpointHook'),
logger=dict(interval=50, log_metric_by_epoch=False, type='LoggerHook'),
param_scheduler=dict(type='ParamSchedulerHook'),
sampler_seed=dict(type='DistSamplerSeedHook'),
timer=dict(type='IterTimerHook'),
visualization=dict(draw=False, type='DetVisualizationHook'))
default_scope = 'mmdet'
dummy_metainfo = dict(classes=('object', ))
env_cfg = dict(
cudnn_benchmark=False,
dist_cfg=dict(backend='nccl'),
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
file_client_args = dict(backend='disk')
launcher = 'none'
load_from = 'checkpoint/UltraSam.pth'
log_level = 'INFO'
log_processor = dict(by_epoch=False, type='LogProcessor', window_size=50)
model = dict(
backbone=dict(
arch='base',
img_size=1024,
out_channels=256,
patch_size=16,
type='mmpretrain.ViTSAM',
use_abs_pos=True,
use_rel_pos=True,
window_size=14),
bbox_head=dict(type='SAMHead'),
data_preprocessor=dict(
bgr_to_rgb=True,
mean=[
123.675,
116.28,
103.53,
],
pad_size_divisor=1024,
std=[
58.395,
57.12,
57.375,
],
type='DetDataPreprocessor'),
decoder=dict(
layer_cfg=dict(
embedding_dim=256,
ffn_cfg=dict(
embed_dims=256, feedforward_channels=2048, ffn_drop=0.1),
num_heads=8),
num_layers=2),
prompt_encoder=dict(
label_encoder=dict(embed_dims=256, type='LabelEmbedEncoder'),
type='SAMPaddingGenerator'),
train_cfg=dict(assigner=dict(type='SAMassigner')),
type='SAM',
use_mask_refinement=True)
optim_wrapper = dict(
clip_grad=dict(max_norm=0.1, norm_type=2),
optimizer=dict(lr=0.0001, type='AdamW', weight_decay=0.0001),
type='OptimWrapper')
orig_test_evaluator = dict(
ann_file=
'UltraSAM_DATA/UltraSAM/MMOTU_2d/annotations/test.agnostic.MMOTU_2d__coco.json',
backend_args=None,
classwise=True,
format_only=False,
metric=[
'bbox',
'segm',
],
type='CocoMetric')
orig_val_evaluator = dict(
ann_file='UltraSAM_DATA/UltraSAM/test.agnostic.noSmall.coco.json',
backend_args=None,
classwise=True,
format_only=False,
metric=[
'bbox',
'segm',
],
type='CocoMetric')
param_scheduler = [
dict(
begin=0, by_epoch=False, end=500, start_factor=0.001, type='LinearLR'),
dict(
begin=0,
by_epoch=False,
end=30000,
gamma=0.1,
milestones=[
20000,
28888,
],
type='MultiStepLR'),
]
resume = False
test_ann_file = 'Dataset_BUSI/annotations/Dataset_BUSI__coco.json'
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
batch_size=1,
dataset=dict(
ann_file='Dataset_BUSI/annotations/Dataset_BUSI__coco.json',
backend_args=None,
data_prefix=dict(img='Dataset_BUSI/images/'),
data_root='datasets',
metainfo=dict(classes=('object', )),
pipeline=[
dict(
file_client_args=dict(backend='disk'),
type='LoadImageFromFile'),
dict(keep_ratio=True, scale=(
1024,
1024,
), type='FixScaleResize'),
dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
dict(
get_center_point=True,
normalize=False,
number_of_points=[
1,
],
test=True,
type='GetPointFromBox'),
dict(
max_jitter=0.0, normalize=False, test=True,
type='GetPointBox'),
dict(prompt_probabilities=[
0.0,
1.0,
], type='GetPromptType'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='PackPointDetInputs'),
],
test_mode=True,
type='CocoDataset'),
drop_last=False,
num_workers=2,
sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
ann_file='datasets/Dataset_BUSI/annotations/Dataset_BUSI__coco.json',
backend_args=None,
classwise=True,
format_only=False,
metric=[
'bbox',
'segm',
],
type='CocoMetric')
test_pipeline = [
dict(file_client_args=dict(backend='disk'), type='LoadImageFromFile'),
dict(keep_ratio=True, scale=(
1024,
1024,
), type='FixScaleResize'),
dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
dict(
get_center_point=True,
normalize=False,
number_of_points=[
1,
],
test=True,
type='GetPointFromBox'),
dict(max_jitter=0.0, normalize=False, test=True, type='GetPointBox'),
dict(prompt_probabilities=[
0.0,
1.0,
], type='GetPromptType'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='PackPointDetInputs'),
]
train_cfg = dict(max_iters=30000, type='IterBasedTrainLoop', val_interval=5000)
train_dataloader = dict(
batch_sampler=dict(type='AspectRatioBatchSampler'),
batch_size=8,
dataset=dict(
ann_file='train.agnostic.noSmall.coco.json',
backend_args=None,
data_prefix=dict(img=''),
data_root='UltraSAM_DATA/UltraSAM',
filter_cfg=dict(filter_empty_gt=True),
metainfo=dict(classes=('object', )),
pipeline=[
dict(
file_client_args=dict(backend='disk'),
type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
dict(prob=0.5, type='RandomFlip'),
dict(keep_ratio=True, scale=(
1024,
1024,
), type='FixScaleResize'),
dict(
normalize=False,
number_of_points=[
1,
],
test=False,
type='GetPointFromMask'),
dict(normalize=False, test=False, type='GetPointBox'),
dict(prompt_probabilities=[
0.5,
0.5,
], type='GetPromptType'),
dict(type='PackPointDetInputs'),
],
type='CocoDataset'),
num_workers=6,
persistent_workers=True,
sampler=dict(shuffle=True, type='InfiniteSampler'))
train_pipeline = [
dict(file_client_args=dict(backend='disk'), type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
dict(prob=0.5, type='RandomFlip'),
dict(keep_ratio=True, scale=(
1024,
1024,
), type='FixScaleResize'),
dict(
normalize=False,
number_of_points=[
1,
],
test=False,
type='GetPointFromMask'),
dict(normalize=False, test=False, type='GetPointBox'),
dict(prompt_probabilities=[
0.5,
0.5,
], type='GetPromptType'),
dict(type='PackPointDetInputs'),
]
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
batch_size=1,
dataset=dict(
ann_file='Dataset_BUSI/annotations/Dataset_BUSI__coco.json',
backend_args=None,
data_prefix=dict(img='Dataset_BUSI/images/'),
data_root='datasets',
metainfo=dict(classes=('object', )),
pipeline=[
dict(
file_client_args=dict(backend='disk'),
type='LoadImageFromFile'),
dict(keep_ratio=True, scale=(
1024,
1024,
), type='FixScaleResize'),
dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
dict(
get_center_point=True,
normalize=False,
number_of_points=[
1,
],
test=True,
type='GetPointFromBox'),
dict(
max_jitter=0.0, normalize=False, test=True,
type='GetPointBox'),
dict(prompt_probabilities=[
0.0,
1.0,
], type='GetPromptType'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='PackPointDetInputs'),
],
test_mode=True,
type='CocoDataset'),
drop_last=False,
num_workers=2,
sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
ann_file='datasets/Dataset_BUSI/annotations/Dataset_BUSI__coco.json',
backend_args=None,
classwise=True,
format_only=False,
metric=[
'bbox',
'segm',
],
type='CocoMetric')
vis_backends = [
dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend'),
]
visualizer = dict(
name='visualizer',
type='PointVisualizer',
vis_backends=[
dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend'),
])
work_dir = './work_dirs/UltraSAM_box_refine_BUSI'

10/21 15:21:59 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
10/21 15:21:59 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) MonkeyPatchHook
(BELOW_NORMAL) LoggerHook

before_train:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook

before_train_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook

before_train_iter:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook

after_train_iter:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_train_epoch:
(NORMAL ) IterTimerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

before_val:
(VERY_HIGH ) RuntimeInfoHook

before_val_epoch:
(NORMAL ) IterTimerHook

before_val_iter:
(NORMAL ) IterTimerHook

after_val_iter:
(NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook

after_val_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_val:
(VERY_HIGH ) RuntimeInfoHook

after_train:
(VERY_HIGH ) RuntimeInfoHook
(VERY_LOW ) CheckpointHook

before_test:
(VERY_HIGH ) RuntimeInfoHook

before_test_epoch:
(NORMAL ) IterTimerHook

before_test_iter:
(NORMAL ) IterTimerHook

after_test_iter:
(NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook

after_test_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook

after_test:
(VERY_HIGH ) RuntimeInfoHook

after_run:
(BELOW_NORMAL) LoggerHook

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loads checkpoint by local backend from path: checkpoint/UltraSam.pth
Traceback (most recent call last):
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmdet/.mim/tools/test.py", line 149, in
main()
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmdet/.mim/tools/test.py", line 145, in main
runner.test()
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1821, in test
self.load_or_resume()
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1699, in load_or_resume
self.load_checkpoint(self._load_from)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmengine/runner/runner.py", line 2127, in load_checkpoint
checkpoint = _load_checkpoint(filename, map_location=map_location)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmengine/runner/checkpoint.py", line 548, in _load_checkpoint
return CheckpointLoader.load_checkpoint(filename, map_location, logger)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmengine/runner/checkpoint.py", line 330, in load_checkpoint
return checkpoint_loader(filename, map_location)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmengine/runner/checkpoint.py", line 347, in load_from_local
checkpoint = torch.load(filename, map_location=map_location)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/torch/serialization.py", line 1529, in load
raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint.
(1) In PyTorch 2.6, we changed the default value of the weights_only argument in torch.load from False to True. Re-running torch.load with weights_only set to False will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
(2) Alternatively, to load with weights_only=True please check the recommended steps in the following error message.
WeightsUnpickler error: Unsupported global: GLOBAL mmengine.logging.history_buffer.HistoryBuffer was not an allowed global by default. Please use torch.serialization.add_safe_globals([mmengine.logging.history_buffer.HistoryBuffer]) or the torch.serialization.safe_globals([mmengine.logging.history_buffer.HistoryBuffer]) context manager to allowlist this global if you trust this class/function.

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.
Traceback (most recent call last):
File "/home/user/anaconda3/envs/UltraSam/bin/mim", line 7, in
sys.exit(cli())
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/click/core.py", line 1462, in call
return self.main(*args, **kwargs)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/click/core.py", line 1383, in main
rv = self.invoke(ctx)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/click/core.py", line 1850, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/click/core.py", line 1246, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/click/core.py", line 814, in invoke
return callback(*args, **kwargs)
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mim/commands/test.py", line 109, in cli
is_success, msg = test(
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mim/commands/test.py", line 288, in test
ret = subprocess.check_call(
File "/home/user/anaconda3/envs/UltraSam/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/user/anaconda3/envs/UltraSam/bin/python3.10', '/home/user/anaconda3/envs/UltraSam/lib/python3.10/site-packages/mmdet/.mim/tools/test.py', 'configs/UltraSAM/UltraSAM_zero_shot/UltraSAM_box_refine_BUSI.py', 'checkpoint/UltraSam.pth', '--launcher', 'none']' returned non-zero exit status 1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions