[GSoC'21] Livestreaming v2

[Sov-trotter]

PR Checklist

If you are contributing to Javis.jl, please make sure you are able to check off each item on this list:

• Did I update CHANGELOG.md with whatever changes/features I added with this PR?
• Did I make sure to only change the part of the file where I introduced a new change/feature?
• Did I cover all corner cases to be close to 100% test coverage (if applicable)?
• Did I properly add Javis dependencies to the Project.toml + set an upper bound of the dependency (if applicable)?
• Did I properly add test dependencies to the test directory (if applicable)?
• Did I check relevant tutorials that may be affected by changes in this PR?
• Did I clearly articulate why this PR was made the way it was and how it was made?

Closes #345
As discussed this makes livestreaming more live, this time by streaming the frames rather than a gif.
Also allows to stream specific frames by passing in a UnitRange

....

conf = setup_stream(:local, 10:12)
render(video; streamconfig = conf)

[codecov]

Codecov Report

Merging #387 (4a1b580) into master (2f9e047) will increase coverage by 0.01%.
The diff coverage is 92.85%.

❗ Current head 4a1b580 differs from pull request most recent head f4290ce. Consider uploading reports for the commit f4290ce to get more accurate results

@@            Coverage Diff             @@
##           master     #387      +/-   ##
==========================================
+ Coverage   96.25%   96.26%   +0.01%     
==========================================
  Files          35       35              
  Lines        1521     1528       +7     
==========================================
+ Hits         1464     1471       +7     
  Misses         57       57              

Impacted Files	Coverage Δ
src/structs/Livestream.jl	100.00% <ø> (ø)
src/javis_viewer.jl	91.83% <90.90%> (+0.62%)	⬆️
src/Javis.jl	96.39% <100.00%> (+0.03%)	⬆️
src/util.jl	97.02% <0.00%> (-0.06%)	⬇️
src/shorthands/JBox.jl	100.00% <0.00%> (ø)
src/action_animations.jl	100.00% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f9e047...f4290ce. Read the comment docs.

[Sov-trotter]

@TheCedarPrince @Wikunia I think this is ready for review

[Wikunia]

To be honest I expected something quite different with #345
Currently this does still precomputes all image frames before starting to stream.
Live streaming should mean that it starts streaming as soon as the first frame got computed.

[Wikunia]

what do you mean by faster? Is it live? 😉

[Wikunia]

what happens if tempdirectory is empty? It just generates the frames but can't use it?

[Sov-trotter]

No tempdir is created before if it's empty (as before)
https://github.com/Wikunia/Javis.jl/pull/387/files/56156371a5657544c632a3cb6faa2b39681e1fe7#diff-763ffddd0fe1ffb533564007035f5947812905676655f155ce1ee4357eb53c5eR275

[Wikunia]

Yeah what I mean is: it should never be empty so we don't need the if. The problem is if we have an mp4 as the pathname without a temp directory but want to stream it will fail, right?

[Wikunia]

I'm unsure what broke when merging it with master. Do you have an idea @Sov-trotter ? Feel free to revert the changes I made.

[Sov-trotter]

Live streaming should mean that it starts streaming as soon as the first frame got computed.

Yeah. I understand. Just taking small steps here. Will get to that too. :)

[Sov-trotter]

Live streaming should mean that it starts streaming as soon as the first frame got computed.

I think we cna talk about how to do this in a cleaner fashion. Because subsequent frames are generated in a serial fashion in a for loop.

I wonder how to have ffmpeg read that. We can obviously overwrite one image again and again but that might cause a race condition.

https://newbedev.com/can-you-stream-images-to-ffmpeg-to-construct-a-video-instead-of-saving-them-to-disk might be of help

[Sov-trotter]

Hey @Wikunia @TheCedarPrince after some more experimentation here's what I found:

• I think it's more reasonable to have the the the livestreaming "semi-live" as currently in this PR or lazy streaming? Not sure if I am using the term "lazy" correctly?

• As I mentioned, there are two ways of making it actually live first is using image2pipe and other is streaming a single image and overwriting it in julia.

• image2pipe works. One work around I found was since it doesn't support -stream_loop we can create our own loop in julia that iterates to large values that creates a new ffmpeg process everytime.

• This works but is a bit laggy on ffplay, while obs works smoothly.

• However I find this to be slow since get_javis_frame is called for every iteration of the stream. Hence using image2pipe here is not a good idea since that will require us to pipe the frames for every iteration hence making it computationally expensive.

• The above point also applies to overwriting the frame image, since we expected some caching of the frames to take place but that doesn't quite happen without using stream_loop.

• Also creating a new image and replacing the previous frame with the new one isn't neat?

What we can do is remove the images from temp directory when cancel_stream is called.