Skip to content

cache.md

Latest commit

 

History

History
74 lines (64 loc) · 3.8 KB

cache.md

File metadata and controls

74 lines (64 loc) · 3.8 KB

Cache System

Targets are being cached after creation, and checked if the target is in cache just before executing a command. There are two types of cache that UMake is using local (filesystem) and remote (minio).

How Cache works

On Save

  • sha1 of the target sources (those that were generated from UMakefile) are being calculated and sha1 of the command itself. All dependecies files (also those that were auto detected) Saved to md-<calculated_hash> file.
  • sha1 of all dependecies are calculated and the just created target is saved to <all_dependecies_hash>/<target_name_hash>

On Load

  • sha1 of the target sources (those that were generated from UMakefile) are being calculated and sha1 of the command itself. Reading md-<calculated_hash> for all the file dependencies
  • calculating sha1 of all of the target dependencies (from the files system) and copying <all_dependencies_hash>/<target_name_hash> to the project directory as it was generated by the command

Local Cache

The local cache is stored in ~/.umake/build-cache.

Remote Cache

umake support uploading the artifacts to a remote server. The more people that work on the same code base the more the remote cache will work. Think of it like that, if someone in the office already build a file then there is no need to build it again. This concept speeds up local builds, since if someone else already built the binary we just download from the cache. It also speeds up CI builds, since in most cases developers compile & test their code locally.

Using minio as a umake remote cache

Minio is a high performance open source object storage. It is compatible with AWS S3. It is very easy to set it up using the official docker image. Connecting it umake is also very simple.

How to run minio for umake

We need to start a minio server and make sure that it exposes a port via docker. Use the following command to spin up the server:

docker run \
  -e MINIO_ACCESS_KEY=umake \
  -e MINIO_SECRET_KEY=umakeumake \
  -p 9000:9000 \
  minio/minio server /data

Note that you need to select a password and a username. In this example it's umake/umakeumake. For more minio configurations refer to the docs. Now that the server is up and running we need to create a bucket for umake. The simplest way to do it is via the minio web interface.

  • Access http://localhost:9000 and user the username and password you selected.
  • Create a new bucket called umake-build-cache

And that's it! we are good to go. Let's test it out using the example in the repo

$ cd ./example
$ umake
 [0.082]  done imports
 [0.000]  done loading graph
 [0.000]  done filesystem scan
 [0.001]  done parsing UMakefile
 gcc -c /home/dn/umake/example/hello.c -o /home/dn/umake/example/*.o
 [0.084] [CACHED] /home/dn/umake/example/*.o
 gcc /home/dn/umake/example/*.o -o /home/dn/umake/example/hello_world
 [0.090] [CACHED] /home/dn/umake/example/hello_world
 [0.000]  done saving graph
 [0.004]  done cache gc
 Workers  0/8 Cache  0/1500[MB]  Cache Hits  0%  Local/Remote  0%/0%   Variant  default  Time   0[sec]  hello_world

 $ umake --remote-cache-stats
 [0.082]  done imports
bucket size 0MB, n_objects 4

$ umake --no-local-cache
 [0.081]  done imports
 [0.000]  done loading graph
 [0.000]  done filesystem scan
 [0.001]  done parsing UMakefile
 gcc -c /home/dn/umake/example/hello.c -o /home/dn/umake/example/*.o
 [0.008] [REMOTE-CACHE] /home/dn/umake/example/*.o
 gcc /home/dn/umake/example/*.o -o /home/dn/umake/example/hello_world
 [0.007] [REMOTE-CACHE] /home/dn/umake/example/hello_world
 [0.000]  done saving graph
 [0.003]  done cache gc
 Workers  0/8 Cache  0/1500[MB]  Cache Hits  100%  Local/Remote  0%/100%   Variant  default  Time   0[sec]  hello_world