[WIP][video][cuda] NVDECCache eviction policy, static implementation

[vkhalidov]

Summary

Adds the notion of statically defined eviction policy to NVDECCache. Eviction policy decides which entry should leave the cache when it's full and a new decoder needs to be created. This PR suggests the following API for an eviction policy:

 - void onInsert(CacheIterator it)      // Called when entry is added
 - void onRemove(CacheIterator it)      // Called when entry is removed
 - CacheIterator selectForEviction()    // Returns iterator to evict
 - bool empty() const                   // Returns true if nothing to evict

Three example policies are provided:

 - LRUEvictionPolicy     // Least Recently Used eviction policy
 - FIFOEvictionPolicy    // First In, First Out eviction policy
 - NoEvictionPolicy      // No eviction, rejects new entries when the cache is full (used before this PR)

LRUEvictionPolicy is proposed as the default eviction policy.

Test Plan

Benchmarks were run on a set of 1200 diverse videos of 132 different resolutions; top 10 resolution buckets are:

  Count Resolution
    200 720x720
    189 720x1280
    180 640x640
    140 720x900
     70 480x480
     59 360x640
     35 640x360
     30 640x800
     21 480x600
     18 720x406

On the plots below, different numbers of frames at different frame steps were extracted from these videos using different numbers of worker threads by decoders using this PR (with eviction) and decoders before this PR (baseline). Throughput in terms of the number of decoded frames per second (FPS) is plotted as a function of the number of worker threads (# threads).

15 decoded frames:

30 decoded frames:

60 decoded frames:

90 decoded frames:

Conclusion

Eviction policy brings advantage in terms of performance and scaling across almost all the settings. However, if the number of decoded frames is too small (e.g. 15 frames at frame step 1) or too big (60 frames at larger frame steps, 90 frames), cache eviction seems to no longer yield advantage.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1243

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.