Unnecessary file reloading issue slow 318

[ktak-007]

No description provided.

[jappeace]

this c code was already significantly better then what an applicant for senior firmware engineer wrote in London 😅

now it's even better, thank you 🙏🏽

[jappeace]

why do we need this check? and what happens if it's not a gzip/runnabble?

[ktak-007]

[ktak-007]

(I can add puml files to this PR)

[jappeace]

oh wow!
hmmm, is it possible to link from the comment to this puml file? It's beneficial.
great work, thanks 🙏🏽

[ktak-007]

https://github.com/ktak-007/keter/blob/add-docs-puml/doc/SlowCopy.puml

[ulidtko]

Decent workaround, and good diagram! Please add it too.

Two comments.

1. This wires in a hard coupling / dependency on bundles being gzipped. Not uncompressed, not bzip2, not xz, not zst.

   • It's OK for now, as keter hardcodes a dependency onto gzip too:
     

     keter/src/Keter/TempTarball.hs

     Line 61 in ee4c968

     let entries = Tar.read $ decompress lbs

   • but will make adding other compression algos more difficult, would that ever become desired.

2. I think we should not decompress while retaining the entire thing in memory (both the archive, and the resulting tar file). It's a no-brainer use-case for conduits, which Keter already depends on. See inline

[ulidtko]

This can get pretty heavy on RAM.

Bundles can get ~200 MB compressed... and more uncompressed... It's a pity to use so much memory just to get a Bool!

And keter already has conduit-extra in dependencies. So it'd be pretty simple:

import Conduit
import Data.Conduit.Zlib as Zlib

isGzip :: FilePath -> IO Bool
isGzip bundle =
  either (const False) (const True) =<< E.try @SomeException (
    runConduitRes $
      Conduit.sourceFile bundle
      .| Gzip.ungzip
      .| sinkNull 
  )

... and will process the file in constant memory, never retaining more than 1 chunk.

[ktak-007]

I found that this example always shows true on gzip files. To check full file loading I've add readSize and countSize steps. It leads to run it in State monad.

[jappeace]

thanks 🙏🏽