Way to have more threads busy

[FlorianHeigl]

Hi,

I'm just finishing up a disk rescue job for a coworker's family pictures...
It was a FAT disk with a destroyed MFT so the best I could get for them was files named by (exif) date.
Running fussel over it to make it into a browsable gallery so they can find their memories.

The stupid part of the whole story is I've fired up my largest server for the restore to do everything in RAM.
photorec - single threaded, using mostly 1/96 threads.
fussel - single threaded, using 1/96 threads, again.

maybe you could leave some hints how to raise the heap sizes & threads to get through the generation faster for large jobs? It's nice to use something build using modern software, but also means it's not as simple to speed up as doing for folder in stuff ; do find folder -exec blah {} + & done or using parallel or whatever.

If I understand it i can send a PR for the docs. (if not being the highest certainty, but I promise to try)

[FlorianHeigl]

I think I found half of the answer.
There's --label gallery.parallel_tasks=NUM - setting this higher had no effect apparently.
I have seen there's 8 python processes running with 16 vcpus, I'm now testing again with 48 and try if that will change from 8 to 24 now...

[FlorianHeigl]

resized

[root@localhost ~]# grep -c ^proc /proc/cpuinfo 
48
[root@localhost ~]# ps -ef | grep -c fussel.py
26

resized again

[root@localhost ~]# grep -c ^proc /proc/cpuinfo 
64
[root@localhost ~]# ps -ef | grep -c fussel.py
34

yes, this seems to be how this (roughly) works...
I had --label gallery.parallel_tasks=64 on the docker command line for all tests but it seems it has no effect, at least the way I had tried it.

[cbenning]

Firstly you might be using some older code as the latest docker image uses ENV vars to configure rather than labels. So you might want to get the latest code.

Secondly you're right I think the dockerfile template for the config is missing a field for that (or has it commented out https://github.com/cbenning/fussel/blob/main/docker/template_config.yml#L17) I will try and fix this when I get time next week if I can.

In the mean-time I believe the default value is actually core_count / 2? https://github.com/cbenning/fussel/blob/main/fussel/generator/config.py#L52

[FlorianHeigl]

I'll check what version I'm running - I presume I ran into some old documentation while searching for the right knob to tune. that would explain why the setting didn't take effect, too. I'll have to restart after adjusting the heap sizes anyway and after a good nap I'll probably figure out the right settings using ENV ;-)
Thanks for the source link, that will a lot.

second: oh! true! note: it was already quite fast after raising the number of vCPUs. I didn't benchmark but much better than before when it just used 8 threads.

core_count / 2 matches with what I see.
I guess the +2 in my grep are the usual thing of it catching itself (what pgrep is made for when you 100% know your process name, which I wasn't sure of)

[FlorianHeigl]

FYI: I'm now running like this. Fussel flies now.
It's still using only about 25% of available CPU, peaking at 50% - but it's really fast anyway.
Putting the workdir of fussel into RAM was the trick (it's super hard to have /var/lib/docker there, plus one would lost it, hard, on reboot). This should also be faster when not using a Ramdisk just over the overlayfs. In my case I always had to run docker system prune between tries because the container would grow so large.
By breaking out _gallery here, it should be better. (It copies to OUTPUT_PATH only at the end of processing, so the container volume normally holds everything)

docker run \
  -v /xxxxx/sorted_jpg:/input:ro \
  -v /yyyy/gallery:/output \
  -v /dev/shm/fussel-static:/fussel/fussel/web/public/static \
  -e INPUT_PATH="/input" \
  -e OUTPUT_PATH="/output" \
  -e OVERWRITE="False" \
  -e PARALLEL="48" \
  -e EXIF_TRANSPOSE="True" \
  -e RECURSIVE="True" \
  -e RECURSIVE_NAME_PATTERN="{parent_album} > {album}" \
  -e FACE_TAG_ENABLE="False" \
  -e WATERMARK_ENABLE="True" \
  -e WATERMARK_PATH="web/src/images/fussel-watermark.png" \
  -e WATERMARK_SIZE_RATIO="0.3" \
  -e SITE_ROOT="/" \
  -e SITE_TITLE="Gallery" \
  -e NODE_OPTIONS="--max-old-space-size=4096" \
  ghcr.io/cbenning/fussel:latest 

Footnotes:

• So PARALLEL still has no effect, I've just put it in there so I got it ready and don't forget
• I'm still bumping up the Node "max-old-space-size" parameter. Also the face matching seems to also be single-threaded, saw that stay around 99% on one core for a long while. No idea if that is simply due to how the recognition needs to be done.

And sorry I can't run proper timed benchmarks, I had forgotten where I stored the images etc and lost many hours finding that and setting up the server etc., noticing I hadn't collect any videos during sorting etc., still running overtime on this originally tiny favour...

[cbenning]

It's been a while, but if you're still interested in this issue try out the v3 branch I'm working on. It will be released soon:

https://github.com/cbenning/fussel/tree/v3

[FlorianHeigl]

Hi, thanks for the reply. I can't test it anymore. FYI I had used fussel to rescue a colleagues family photo library. it was on a crashed disk with broken FAT. using testdisk and imagerec (??) i managed to scrap off all images and sort them by date based off the exif data. and fussel built a browsable gallery so they could find stuff again. did the whole thing on a dual xeon E7 where it basically fit in RAM. but i deleted my work copies afterwards so atm i can't run a similar test. but any improvement will help the next person doing a similar thing :-) Thanks for your work, it's a great tool! Florian Chris Benninger ***@***.***> schrieb am Mi., 28. Jan. 2026, 04:47:

…

*cbenning* left a comment (cbenning/fussel#144) <#144 (comment)> It's been a while, but if you're still interested in this issue try out the v3 branch I'm working on. It will be released soon: https://github.com/cbenning/fussel/tree/v3 — Reply to this email directly, view it on GitHub <#144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVSVGYC5T2RPAFYQD6NT4L4JAWLXAVCNFSM6AAAAACBMG54RSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQMBYG44DONRZGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>