oshamash · oshamash · Feb 27, 2020 · Feb 27, 2020 · Feb 28, 2020 · Feb 29, 2020
diff --git a/.drone.yml b/.drone.yml
@@ -0,0 +1,27 @@
+---
+kind: pipeline
+name: default
+type: docker
+
+steps:
+  -
+    name: tests
+    commands:
+      # /umake holds the binaries which are installed in the image.
+      # refer to the dockerfile.
+      - "pip3 install --upgrade ."
+      - "cd umake && pyflakes ."
+      - "cd ../test && python3.6 test.py"
+    image: grisha85/umake:3
+    pull: if-not-exists
+
+  - name: coverage
+    image: plugins/codecov
+    settings:
+      token:
+        from_secret: CODECOV
+      required: true
+      files:
+        - "*.xml"
+      paths:
+        - test/coverage
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,9 @@
 test/env/*
 __pycache__
+test/coverage/*
+test/.coverage
+example/hello.o
+example/.umake
+example/hello_world
+example/.coverage
+umake.egg-info
diff --git a/Dockerfile b/Dockerfile
@@ -1,9 +1,12 @@
 FROM ubuntu:18.04
 RUN apt-get update -y
-RUN apt-get install -y python3.6 build-essential python3-pip libxml2-dev zlib1g-dev strace
-RUN apt-get install -y git
+RUN apt-get install -y python3.6 build-essential python3-pip libxml2-dev zlib1g-dev strace vim wget
 ADD . /umake
-RUN pip3 install  /umake
+RUN pip3 install -e /umake
+
+RUN pip3 install ipdb coverage pyflakes
+
+RUN wget https://dl.min.io/server/minio/release/linux-amd64/minio && chmod +x ./minio && mv ./minio /usr/bin
 
 # for tests
 RUN apt-get install -y libprotobuf-c0-dev protobuf-c-compiler
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,25 @@
+BSD 2-Clause License
+
+Copyright (c) 2020, Gregory Freilikhman
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/Makefile b/Makefile
@@ -0,0 +1,22 @@
+
+
+.PHONY: test enter benchmark
+
+IMAGE=grisha85/umake:3
+
+VOLUMES=-v`pwd`:/umake
+
+build-docker:
+	docker build -t ${IMAGE} .
+
+test: 
+	docker run --rm --privileged -it ${VOLUMES} -w/umake ${IMAGE} bash -c 'cd test && python3.6 test.py'
+
+lint:
+	docker run --rm ${VOLUMES} -w/umake/umake ${IMAGE} pyflakes .
+
+enter:
+	docker run --rm --privileged -it ${VOLUMES} -w/umake ${IMAGE} bash 
+
+benchmark:
+	docker run --rm --privileged -it -v`pwd`:/umake -w/umake ${IMAGE} bash -c 'cd test && python3 ./test.py TestUMake.test_benchmark'
diff --git a/README.md b/README.md
@@ -1,213 +1,86 @@
 UMake
 =====
-**Blazing Fast. Sub scecond Modification detection. Few seconds for cached compilation**
+**Blazing Fast. Sub second Modification detection. Just a few seconds for cached compilation**
+
+![dpdk build](doc/images/dpdk-build/dpdk-build.gif)
 
 Overview
 --------
-UMake is a build system that building your projects.  
-influenced by [`tup`](http://gittup.org/tup/). 
+UMake is a build system that building your projects.
+Influenced by [`tup`](http://gittup.org/tup/). With the features below your compilation speed will be on average dramatically reduced, either after branch change either with your `CI` builds.
 
-* local cache - disk cache
-* remote cache - minio
-* auto dependency discovery using strace
-* simple configuration language
+[![codecov](https://codecov.io/gh/grisha85/umake/branch/master/graph/badge.svg)](https://codecov.io/gh/grisha85/umake/tree/master/umake)
 
-Running example
----------------
 
-```
-git clone https://github.com/grisha85/umake.git
-cd umake
-docker build -t umake  .
-docker run --rm -it -v`pwd`/example:/example -w/example umake bash
-umake --no-remote-cache
-./hello_world
-```
+[![droneio](http://xrayio.com/api/badges/grisha85/umake/status.svg)](http://xrayio.com/grisha85/umake/)
 
-How UMake works
----------------
-* loading compilation graph (DAG) from previous build (graph is empty when built first time)
-
-* scannig filesytem using the loaded graph to check for changes
 
-  * `modified` - if file was modified on filesystem it marked as modified on the graph
-  * `deleted` - if file was deleted from the filesystem, successor target deleted as well from the filesystem and the graph
-
+#### `Local cache`
+many base libraries in your project rarely changed, why recompile them over and over again. Local cache reduce compilation times and remote cache access.
 
-* parsing UMakefile and creating new commands or updating the existing
+#### `Remote cache`
+If someone already compiled most of the libraries in your project, use those results.
 
-  * `deleted` - if command deleted, targets of this command deleted as well from both graph and filesystem
-  * `updated` - if command is updated, UMake will handle it as `delete` and `create`. so, target of the old command will be deleted and new target will be created when graph will be executed
-* executing the graph in parallel 
+#### `Auto dependency discovery`
+makes your life easier to create build scripts no matter what your tool is: `gcc`, `protoc`, `docker build` ...
 
-  * `auto dependency detection` - updating the graph with accessed files by parsing strace logs
-  * `cache` - saving to cache. more details: [Cache System](#cache-system)
-* saving the build graph 
 
-Note: by automatically deleting `targets` when no longer needed (either `command` is delete or source file was the deleted for this `target`) UMake implementing `clean` by design. So `clean` no longer need to be mainained.
+[Detailed overview](doc/overview.md)
 
-UMakefile
----------
-## Rule `:`
+[Cache System](doc/cache.md)
 
-A signle `command` is generated for this rule
+[UMakefile](doc/umakefile.md)
 
-`:` source | manual-deps `>` cmd `>` target
+Install
+-------
 
-`manual-deps` - targets the this tule depends on, in order to keep a correct build order  
-`cmd` - bash command  
-`target` - the output of the command  
-
-`{filename}` - full path filename of the source `/my/path/filename.a`  
-`{dir}` - directory of the source `/my/path/`  
-`{noext}` - filename without extension `filename`  
-`{target}` - expanded target `helloworld.a`  
+platform: linux (tested on ubuntu 18.04)
 
-Example:
-```
-: *.o > gcc {filename} -o {target} > helloworld.a
-```
+dependencies: strace, bash. python3
 
-#### Recursive Source `**`
-recursice deps are support
-```
-root\
-  a\
-    a.a\
-      a.a.a
-      a.a.b
-      a.b.a
-    a.b\
-      a.b.a
-      a.b.b
-  b\
-    b
-```
-* `root/**` -> (`a.a.a`, `a.a.b`, `b`)
-* `root/a/**/*.b` -> (`a.a.b`, `a.b.b`)
+ubuntu packages(apt-get install): build-essential python-dev libxml2 libxml2-dev zlib1g-dev
 
-#### Manual Dependency `|`
-In order to maintain the correct order of the build (that is done in parallel), there are use cases that manual depndecy is needed to be provided. for example: if there are `generated headers` that used later by other `command` to generate anoter target. 
+for more details check the `Dockerfile` how to create environment for umake.
 
-
-## Rule `:foreach`
-Same as `:` but will create `command` for each `source` (that match the pettern *.o in the example above) file that will be found on the filesystem
-
-## Macro `!`
-Macros are expanded immediatlly (like `#define X "hello"` in c/cpp)  
-Macros can accept parameters
-
-Example:
-```
-!c(includes, flags) : gcc -g -O2 -Wall -fPIC -c {filename} $includes $flags -o {target} > {dir}/{noext}.o  
-```
-#### Default values
-`Macro` supports defualt values, by default it `""`:
-```
-!c(includes, flags=-O3) : gcc -g -O2 -Wall -fPIC -c {filename} $includes $flags -o {target} > {dir}/{noext}.o  
-```
-now `!c` can be called as following
-```
-!c(-Iinclude)       # includes = -Iinclude, flags=-O3
-!c(-Iinclude, -O0)  # includes = -Iinclude, flags=-O0
-!c()                # includes = "", flags=-O3
 ```
-## Const `$`
-Consts are like macros, and can be used to parametrize calls to macros
-Example:
-```
-$libs = -lpthread
-!so($libs)
-```
-
-## Config `[<config_item>:<config_value>]`
-Configs allow to configure and changing umake execution.
-
-#### `workdir`
-Default: \<root>
-
-Changing current working directory. After changing the working directory all `relative paths` are relative to the new working dir. `Absoulte paths` are relative to the `root` (the directory where UMakefile exists).
-Relative path `my_dir_a/my_dir_b` will be evaluated as `<workdir>/my_dir_a/my_dir_b`. However `/my_dir_a/my_dir_b` will be evaluated as `<root>/my_dir_a/my_dir_b` regardless what working dir is.
-
-The following rules are similar:
-
-```
-: src/packages/a > gcc > src/packages/b
-```
-```
-[workdir:src/packages]
-: a > gcc > b
-```
-Return to root
-```
-[workdir:/]
-```
-
-#### `variant`
-
-Defult: "default"
-
-The ability to generate diffrent variants from the same sources. For example: debug/release compilations. variant `terminated` with a `newline`
-```
-# varaint is terminated with newline
-[variant:default]
-$cflags = -O3
-
-[variant:debug]
-$cflags = -O0
-
-: my.c > !c($cflags) > my.o
-```
-now compile with `umake` for default variant 
-```
-umake
-```
-or
-```
-umake --variant debug
+git clone https://github.com/grisha85/umake.git
+cd umake
+pip3 install .
 ```
-for `debug` variant.
-
-#### `include`
-Default: -
 
-include another `UMakefile` into the current one.
+Running example
+---------------
 ```
-[include:somedir/umakefile]
+git clone https://github.com/grisha85/umake.git
+cd umake
+docker build -t umake  .
+docker run --rm -it -v`pwd`/example:/example -w/example umake bash
+umake --no-remote-cache
+./hello_world
 ```
-will open and parse `somedir/umakefile` in the current working dir context.
-# Cache System
-Targets are being cached after creation, and checked if the target is in cache just before executing a `command`. There are two types of cache that UMake is using local(filesystem) and remote (minio). 
 
-## How Cache works
-### On Save
-* `sha1` of the target sources (those that were generated from UMakefile) are being calculated and `sha1` of the `command` itself. All dependecies files (also those that were auto detected) Saved to `md-<calculated_hash>` file.
-* `sha1` of all dependecies are calculated and the just created target is saved to `<all_dependecies_hash>/<target_name_hash>`
-### On Load
-* `sha1` of the target sources (those that were generated from UMakefile) are being calculated and `sha1` of the `command` itself. Reading `md-<calculated_hash>` for all the file dependecies
-* calculating `sha1` of all of the target dependecies (from the files system) and copying `<all_dependecies_hash>/<target_name_hash>` to the project directory as it was generated by the `command`
+UMake configuration
+-------------------
+This section lists all the various configurations that umake supports
 
-## Local Cache
-The local cache is stored in `~/.umake/build-cache`. 
+| Variable name                  | Description                                                |
+|--------------------------------|------------------------------------------------------------|
+| UMAKE_CONFIG_ROOT              | The root directory in which all umake files will be stored |
 
-## Remote Cache
-TBD
-# Arguments
+Real Life Examples
+------------------
+[DPDK build](doc/dpdk-build.md)
 
-```
-usage: umake [-h] [--details] [--json JSON_FILE] [--no-remote-cache]
-             [--no-local-cache]
-             [target]
-
-positional arguments:
-  target             target path
 
-optional arguments:
-  -h, --help         show this help message and exit
-  --details          details about the target
-  --json JSON_FILE   output as json
-  --no-remote-cache  don't use remote cache
-  --no-local-cache   don't use local cache
+Talking about UMake:
+--------------------
+This section includes link to various places around the web that reason about umake.
+We believe that by reviewing questions and opinions that other people wrote about umake one can learn more about it.
+So without further ado is here is the list:
 
-```
+* [DriveNets blog](https://drivenets.com/blog/technology-developments/the-inside-story-of-how-we-optimized-our-own-build-system/)
+* [Reddit r/bazel](https://www.reddit.com/r/bazel/comments/fa084s/how_we_optimised_our_build_system_using_umake/)
+* [Reddit r/cpp](https://www.reddit.com/r/cpp/comments/f9yjxn/how_we_optimised_our_build_system_using_umake/)
+* [Reddit r/gcc](https://www.reddit.com/r/gcc/comments/faiqum/how_we_optimised_our_build_system_using_umake/)
 
+Have another story to share about umake? just open a PR with a change to this list and we'll merge it in.
diff --git a/UMakefile b/UMakefile