Feature Request: Caching

[eterry1388]

We have a large codebase of 1,277 erb files. Running 12 linters on all of those files, it takes over 3 minutes. This is a bottleneck to our build system. Ideally, we could run these in parallel or with caching (so only changed files are inspected). This problem will only get worse as we enable more linters and add more erb files to our codebase.

Rubocop has a cache option explained here: https://github.com/rubocop-hq/rubocop/blob/master/manual/caching.md

We would love if such a feature existed for erblint. Thank you!

[heqianw]

The documentation link was moved to this current location: https://github.com/rubocop-hq/rubocop/blob/master/docs/modules/ROOT/pages/usage/caching.adoc

[zachfeldman]

@eterry1388 an initial caching implementation is complete in #268! If you, @heqianw , or anyone else following this thread want to give it a review and/or try out the branch, I'm definitely game for comments 👍

[joshuapinter]

@zachfeldman Incredible! Thanks for putting in the work to get this done. I'll use your fork and see how it goes at CNTRAL.

[joshuapinter]

Results:

Before

$ time be rake default:erblint

====
Running ERB Lint...

Linting 596 files with 9 linters...
No errors were found in ERB files
bundle exec rake default:erblint  72.19s user 3.06s system 95% cpu 1:18.75 total

After

$ time be rake default:erblint

====
Running ERB Lint...

Cache mode is on
Linting 596 files with 9 linters...
No errors were found in ERB files
bundle exec rake default:erblint  71.87s user 3.09s system 95% cpu 1:18.28 total

$ time be rake default:erblint

====
Running ERB Lint...

Cache mode is on
Linting 596 files with 9 linters...
No errors were found in ERB files
bundle exec rake default:erblint  71.52s user 3.00s system 95% cpu 1:17.74 total

$ time be rake default:erblint

====
Running ERB Lint...

Cache mode is on
Linting 596 files with 9 linters...
No errors were found in ERB files
bundle exec rake default:erblint  3.01s user 2.70s system 64% cpu 8.896 total

NOTE: I had to run it 3 times with caching mode on before I saw the increase speed. My assumption was that it would take running it only once with caching mode on for the files to be cached and then the second time would make use of that cache and run quickly. Maybe @zachfeldman can explain why this is.

For anybody else wanting to try @zachfeldman's fork, here's what you need to do:

1. Add his fork to your Gemfile like this:

   gem "erb_lint", 
     github: "zachfeldman/erb-lint",
     branch: "zfeldman/cbruckmayer/implement-file-cache",
     require: false

2. Use --with-cache when running the erblint command.

3. Run the command a few times with caching mode turned on before you see the benefits.

Question @zachfeldman: Where is the cache stored? In development this works out of the box but I'll want to use this in our Github Actions CI and will likely need to cache and restore it specifically in between setups.

Thanks again for this. Seriously awesome. Makes the need for multithreading moot.

[joshuapinter]

Question @zachfeldman: Where is the cache stored? In development this works out of the box but I'll want to use this in our Github Actions CI and will likely need to cache and restore it specifically in between setups.

Never mind, it's cached to .erb-lint-cache directory. I updated our CNTRAL CI to cache and restore this directory automatically to keep things fast. It'll take a few runs before all of our Runners' caches are warm.

I also added this to .gitignore. 👍

[eterry1388]

Amazing work! Here are my results (on a different codebase than from 2 year ago when I originally posted this feature request). My current .erb-lint.yml file looks like this:

linters:
  Rubocop:
    enabled: true
    rubocop_config:
      inherit_from:
        - .rubocop.yml

On current Shopify:main

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all
Linting 464 files with 1 linters...

No errors were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all  22.66s user 0.90s system 89% cpu 26.343 total

On zachfeldman:zfeldman/cbruckmayer/implement-file-cache

First run

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 464 files with 1 linters...

No errors were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   22.75s user 0.99s system 82% cpu 28.796 total

Second run

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 464 files with 1 linters...

No errors were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   1.42s user 0.80s system 47% cpu 4.691 total

Great improvement! Went from over 22 seconds to less than 2 seconds.

I checked the created .erb-lint-cache directory and saw the checksum files:

ls .erb-lint-cache | wc -l
     464

Finally, I tested this with files that had errors

First run

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 464 files with 1 linters...

107 error(s) were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   22.79s user 1.01s system 82% cpu 28.869 total

Second run

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 464 files with 1 linters...

107 error(s) were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   4.00s user 1.04s system 54% cpu 9.273 total

I then fixed some of the errors and ran with --with-cache to make sure even with cache the errors would be resolved:

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 464 files with 1 linters...

104 error(s) were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   3.86s user 0.92s system 55% cpu 8.580 total

Which it did!

Some of the cache files do get large:

ls -alhS .erb-lint-cache
total 7968
-rw-r--r--    1 ericterry  staff    32K Sep 27 07:45 a708b91b7d5ab225f5dc30980a73f9c4472e7921
-rw-r--r--    1 ericterry  staff    29K Sep 27 07:45 b21aeb1092895e1d364dd06d7b98e201bf62856f
-rw-r--r--    1 ericterry  staff    26K Sep 27 07:45 dc8cab6f33edcf2a527781e7e2cf8a84d85ff286
-rw-r--r--    1 ericterry  staff    22K Sep 27 07:45 41ac9e4be5242d99247bae207463e125abb52076

But I assume this is expected and probably similar to what rubocop does.

[eterry1388]

Wanted to post one more test, this time with a ridiculous number of errors.

TL;DR

• Using cache when there are little to no errors greatly improves the speed of the run
• Using cache when there are a lot of errors greatly reduces the speed of the run, consumes a large amount of memory, and makes for very large cache files

First run

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 461 files with 18 linters...

14206 error(s) were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   29.23s user 1.24s system 84% cpu 36.071 total

Second run

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all --with-cache
Cache mode is on
Linting 461 files with 18 linters...

14206 error(s) were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all   331.17s user 15.63s system 68% cpu 8:28.47 total

Yikes! With caching made it a LOT longer to run. I check out out the cache directory and some of the files were massive:

ls -alhS .erb-lint-cache
total 103232
-rw-r--r--    1 ericterry  staff   855K Sep 27 08:02 965f5b1d3d0c976a6ee368ac55d881185a8f306f
-rw-r--r--    1 ericterry  staff   609K Sep 27 08:01 0135f4d6d8b53ee8b87e1688489261ad09f98b19
-rw-r--r--    1 ericterry  staff   568K Sep 27 08:02 a2e1c15703c939c395640a39027aae4f83238cd0
-rw-r--r--    1 ericterry  staff   564K Sep 27 08:01 df23b118633784d2179883912dda76a75dc1a02c
...

This second run was using consistent high CPU the whole time (looking at htop) and the memory usage continued to grow slowly up to around 3.5 GB!

I wanted to run a third time, but this time without --with-cache to make sure it was an issue with this new feature as opposed to just a general feature with erblint

time be bundle exec ../../personal/erb-lint/exe/erblint --lint-all
Linting 461 files with 18 linters...

14206 error(s) were found in ERB files
bundle exec bundle exec ../../personal/erb-lint/exe/erblint --lint-all  27.94s user 0.87s system 89% cpu 32.256 total

It worked fine and memory usages stayed low.

[zachfeldman]

@joshuapinter @eterry1388 thanks so much for trying the cache and posting your results here! Super useful data to see how it works in other projects. As I mentioned in the pull request I unfortunately haven't had time to return to this as I'm only working on it in my spare time, but hope to soon.