Wrong file size after splitting into tracks

[bobmvg]

Hello.
Thank you for the excellent program!
I use it to distribute a small music collection (FLAC Image+CUE) via DLNA (minidlna) for playback on LG TV.
Mostly everything works fine, but after dividing some albums into tracks, trackfs reports the wrong file sizes for individual tracks.

Here is the track information from the Fuse file system and the mediainfo output for one of the tracks:

server /mnt/raid/Музыка/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) # ls -l
total 0
-rw-rw-r-- 1 p2p p2p     2989 дек 10  2022 'Dark Side Of The Moon.log'
-rw-rw-r-- 1 p2p p2p     1062 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.cue'
-rw-rw-r-- 1 p2p p2p 31834908 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.001.Speak To M.flac'
-rw-rw-r-- 1 p2p p2p 37622592 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.002.On The Run.flac'
-rw-rw-r-- 1 p2p p2p 75372192 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.003.Time.flac'
-rw-rw-r-- 1 p2p p2p 50169924 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.004.The Great .flac'
-rw-rw-r-- 1 p2p p2p 69222888 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.005.Money.flac'
-rw-rw-r-- 1 p2p p2p 81401544 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.006.Us And The.flac'
-rw-rw-r-- 1 p2p p2p 36359568 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.007.Any Colour.flac'
-rw-rw-r-- 1 p2p p2p 40681368 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.008.Brain Dama.flac'
-rw-rw-r-- 1 p2p p2p 21928284 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.009.Eclipse.flac'
drwxrwxr-x 2 p2p p2p     4096 дек 10  2022  Диск
server /mnt/raid/Музыка/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) #
server /mnt/raid/Музыка/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) # mediainfo Pink\ Floyd\ -\ Dark\ Side\ Of\ The\ Moon.flac.009.Eclipse.flac 
General
Complete name                            : Pink Floyd - Dark Side Of The Moon.flac.009.Eclipse.flac
Format                                   : FLAC
Format/Info                              : Free Lossless Audio Codec
File size                                : 20.9 MiB
Duration                                 : 2 min 4 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 1 410 kb/s
Album                                    : Dark Side Of The Moon
Album/Performer                          : Pink Floyd
Track name                               : Eclipse
Track name/Position                      : 9
Track name/Total                         : 9
Performer                                : Pink Floyd
Composer                                 : Pink Floyd
DISCID                                   : 6F0A1509

Audio
Format                                   : FLAC
Format/Info                              : Free Lossless Audio Codec
Duration                                 : 2 min 4 s
Bit rate mode                            : Variable
Bit rate                                 : 1 410 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Bit depth                                : 16 bits
Compression mode                         : Lossless
Stream size                              : 20.9 MiB (100%)
Writing library                          : libFLAC 1.3.3 (2019-08-04)
MD5 of the unencoded content             : 40F74FD99EC4C887785C3947D1DFEDF0



server /mnt/raid/Музыка/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) #

This file gives an error when played (here is part of the minidlna log):

[2023/10/05 07:11:35] minidlna.c:170: debug: HTTP connection from 192.168.2.56:56860
[2023/10/05 07:11:35] upnphttp.c:264: debug: Range Start-End: 12713984 - -1
[2023/10/05 07:11:35] clients.c:357: debug: Client found in cache. [LG/entry 0]
[2023/10/05 07:11:35] upnphttp.c:916: debug: HTTP REQUEST: GET /MediaItems/39384.flac HTTP/1.1
Host: 192.168.2.1:8200
User-Agent: Mozilla/5.0 (Web0S; Linux/SmartTV) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 WebAppManager
Connection: close
icy-metadata: 1
Range: bytes=12713984-


[2023/10/05 07:11:35] upnphttp.c:1998: info: Serving DetailID: 39384 [/mnt/raid/Музыка/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  )/Pink Floyd - Dark Side Of The Moon.flac.009.Eclipse.flac]
[2023/10/05 07:11:35] upnphttp.c:2064: warn: Specified range was invalid!
[2023/10/05 07:11:35] upnphttp.c:1277: debug: HTTP RESPONSE: HTTP/1.1 400 Bad Request
Content-Type: text/html
Connection: close
Content-Length: 136
Server: 6.1.41-gentoo DLNADOC/1.50 UPnP/1.0 MiniDLNA/1.3.3
Date: Thu, 05 Oct 2023 04:11:35 GMT
EXT:

<HTML>
    <HEAD>
        <TITLE>400 Bad Request</TITLE>
    </HEAD>
    <BODY>
        <H1>Bad Request</H1>The request is invalid for this HTTP version.</BODY>
</HTML>

If I copy the same files from the Fuse file system to a real hard drive, then the copied files are smaller in size, mediainfo shows a smaller size and a different bitrate:

server /mnt/1/torrents/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) # ls -l
total 265476
-rw-rw-r-- 1 p2p p2p     2989 дек 10  2022 'Dark Side Of The Moon.log'
-rw-rw-r-- 1 p2p p2p     1062 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.cue'
-rw-rw-r-- 1 p2p p2p 23214518 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.001.Speak To M.flac'
-rw-rw-r-- 1 p2p p2p 21382110 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.002.On The Run.flac'
-rw-rw-r-- 1 p2p p2p 44669457 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.003.Time.flac'
-rw-rw-r-- 1 p2p p2p 26696409 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.004.The Great .flac'
-rw-rw-r-- 1 p2p p2p 46425658 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.005.Money.flac'
-rw-rw-r-- 1 p2p p2p 49211303 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.006.Us And The.flac'
-rw-rw-r-- 1 p2p p2p 22555369 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.007.Any Colour.flac'
-rw-rw-r-- 1 p2p p2p 25073379 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.008.Brain Dama.flac'
-rw-rw-r-- 1 p2p p2p 12587210 дек 10  2022 'Pink Floyd - Dark Side Of The Moon.flac.009.Eclipse.flac'
drwxrwxr-x 2 p2p p2p     4096 дек 10  2022  Диск
server /mnt/1/torrents/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) #
server /mnt/1/torrents/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) # mediainfo Pink\ Floyd\ -\ Dark\ Side\ Of\ The\ Moon.flac.009.Eclipse.flac 
General
Complete name                            : Pink Floyd - Dark Side Of The Moon.flac.009.Eclipse.flac
Format                                   : FLAC
Format/Info                              : Free Lossless Audio Codec
File size                                : 12.0 MiB
Duration                                 : 2 min 4 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 809 kb/s
Album                                    : Dark Side Of The Moon
Album/Performer                          : Pink Floyd
Track name                               : Eclipse
Track name/Position                      : 9
Track name/Total                         : 9
Performer                                : Pink Floyd
Composer                                 : Pink Floyd
DISCID                                   : 6F0A1509

Audio
Format                                   : FLAC
Format/Info                              : Free Lossless Audio Codec
Duration                                 : 2 min 4 s
Bit rate mode                            : Variable
Bit rate                                 : 809 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Bit depth                                : 16 bits
Compression mode                         : Lossless
Stream size                              : 12.0 MiB (100%)
Writing library                          : libFLAC 1.3.3 (2019-08-04)
MD5 of the unencoded content             : 40F74FD99EC4C887785C3947D1DFEDF0



server /mnt/raid/Музыка/Pink Floyd - Dark Side Of The Moon 1973 ( SomeWax 2002г.  ) #

These files play normally.

Here is a link to the original files to check (FLAC Image+CUE):
https://drive.google.com/file/d/11Dp453WCW6Lk3jD_7Cwiku6pJ2_2Tocp/view?usp=sharing

[andresch]

Hi bobmvg,

Thanks for using trackfs.

The wrong file size is a limitation of the fundamental design of trackfs. The virtual filesystem you see doesn't exits anywhere on disk, but is created on the fly by reading directories and metadata from the flac+cue files in the source filesystem. Only when an application opens one of the virtual track files in this virtual filesystem, trackfs extracts an individual track from the source flac+cue file (by recoding a part of the source file) in a temporary file. And only after that I would know how big the track file actually is.

The same applies to entries of the virtual directories. They file information for all entries of a directory are generated on the fly from the source directory by adding additional entries for each track I find in track+cue files in the source directory. For each track entry I have to provide some size information. The current implementation always provides a value that corresponds to the size of a non-compressed audio file. Unfortunately I did not find a way to reliably predict the flac compressed size of the track.

see

trackfs/trackfs/flactracks.py

Line 349 in 23b186b

def estimate_track_file_size(self, path: os.PathLike, fp: FusePath) -> int:

In my various experiments I never had issues with too big file size values, only with too little ones -- hence my decision to go for a upper bound. While this always worked for me and my audioplayer, it obviously it doesn't work for you.

If I would have to re-encode all individual tracks before I could respond to directory list requests, then this would take ages and make trackfs filesystem unusable.

Unfortunately I have no reasonable idea how to overcome this problem without changing the fundamental, low-tech architecture of trackfs: Rather that doing everything on the fly, trackfs would during initialization have to once recode all tracks and then store the size values in some database. This would not only significantly change the complexity of my small implementation. For large music collection such an initialization would also run for hours, while now trackfs is ready to run within milliseconds.

The only solution that would come to my mind for your setup is to do some automatic copy/backup of the virtual file system to some physical file system. As you've seen, the copy operation of your operating system can deal with the wrong file size easily. Then the files on that file system would have the actual file size; but of course you would then also need twice as much disk space. Not nice; but maybe still acceptable.

I do not know exactly why mediainfo gives different bit rates on the two files. I would guess the bitrate is not stored directly in the file's metadata, but just calculated by dividing the file size (from the file-system) with the track duration. At least that would explain the difference.

I am sorry for not being able to provide a solution for your problem with trackfs directly. Would be curious to learn if you found a way that works for you. Might be interesting for others that run into similar issues.

Andreas

BTW: When I initially wrote trackfs I was hoping to find some means to directly read the already encoded track from the source file. My hope was that this would allow me to improve the track read performance significantly and could then I could most likely also directly calculate the size of the track from the source file. But unfortunately I never got a response for the creators of the flac encoding on my question if this would be possible. Hence I currently have no alternative to extract and recode each individual track.

[bobmvg]

Hi Andreas!

Thanks for the detailed answer!
As I understand it, in this case it all depends on the DLNA client whether it will continue playback if an error occurs when requesting a non-existent part of the track.
The standard media player for LG TV does not play them.
On PC, such tracks can be played normally through VLC and Kodi and cannot be played through Jellyfin.
Unfortunately, I did not find a universal player in the official applications for WebOS, but quite by accident I came across the version of Kodi for WebOS - https://kodi.wiki/view/HOW-TO:Install_Kodi_for_webOS !
It appeared quite recently, is still in the alpha version stage and is not very stable, but now it allows you to play music without problems as individual tracks from IMAGE, both through the CUE file directly and through trackfs.
It works fine connecting via NFS, SMB and DLNA.
I really hope that it will appear in official applications for WebOS.

[andresch]

As I understand it, in this case it all depends on the DLNA client whether it will continue playback if an error occurs when requesting a non-existent part of the track.

Correct. Given the error you have provided above, I assume that in this case the client tries to (pre-)load a specific range of the file (that doesn't exist) based on the given wrong size of the file, rather than just reading the file sequentially in one go and stop reading when it receives an end-of-file indicator.