Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
dfc62c4
Add a 1 line description at the top of README.md
a-magdy Jul 28, 2025
2f58246
Add a 1 line description at the top of README.md
a-magdy Jul 28, 2025
c1aece2
Update README.md to enhance clarity on benefits and rationale for usi…
a-magdy Jul 28, 2025
ccbab0b
Update the 1 line description at the top of the git-artifact
a-magdy Jul 28, 2025
a13c0b3
minor linting in git-artifact and _tests.sh
a-magdy Jul 28, 2025
9da1e40
Refactor the first 2 sections in README.md to answer the 2 questions …
a-magdy Jul 28, 2025
8c54b51
Remove a second from README.md as it is now included in the Benefits …
a-magdy Jul 28, 2025
916d4ae
Update the structure of README.md to get into how to use it faster
a-magdy Jul 28, 2025
a065190
Use copilot to proof read the README.md file
a-magdy Jul 28, 2025
037e478
minor linting update in git-artifact
a-magdy Jul 28, 2025
0f0b2be
Replace the Super simple block with Quick Start (TL;DR)
a-magdy Jul 28, 2025
a2f96c2
Shorten the Quick Start
a-magdy Jul 28, 2025
b52a060
Move Notes section to above Advanced
a-magdy Jul 28, 2025
4101643
Fix the git artifact init command and cd into the folder afterwards
a-magdy Jul 28, 2025
5073a91
Update the install git-artifact section in the README
a-magdy Jul 29, 2025
1710f38
Update Quick Start (TL;DR) section to be a code block that can be cop…
a-magdy Jul 29, 2025
ed03a6b
Revert changes to the set_opts_spec
a-magdy Jul 29, 2025
e592284
Fix mermaid code block formatting in README
a-magdy Jul 29, 2025
917c9ff
Move Key benefits to above quick start
a-magdy Jul 29, 2025
0e9763a
Use git clone in the README.md
a-magdy Jul 29, 2025
230d478
Help should be called only using (-h) for now (--help) will be suppor…
a-magdy Jul 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
262 changes: 156 additions & 106 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,155 @@
# git-artifact
# git-artifact 🚀📦

## The rational for storing artifacts in git
> Effortless artifact management using Git repositories

I have, over the years in the embbeded enterprise industry, constantly come across many scenarios where zipping, downloading and unzipping generic dependencies and maintaining workspace
has slowed down turn around time for developers and CI system. Git is a fantastic zipper it self and you get integrity of workspaces for free.
## Key Benefits

Git has always been mentioned to be bad for storing artifacts due to the block chain technology and distrubuted architecture. `git-artifact` makes sure this problem is handled by storing commits "horisontally" using tags rather than the default "stacked" way. It gives a few advantages compared to standard usage of git:
- Firstly; You can garbage collect intermidiate artifacts by just deleting the tag
- Secondly; You only fetch what you need - even without using shallow.
- **Seamless integration:** Manage artifacts alongside your source code using familiar Git tools.
- **Efficient storage:** Artifacts are stored as independent commits, so you fetch only what you need.
- **Traceability & integrity:** Tags and Git’s checksums provide clear versioning and authenticity.
- **Easy cleanup:** Remove intermediate artifacts by simply deleting their tags.
- **Unified workflow:** Eliminate the need for separate artifact repositories or complex tools.

### CI/CD integration
Whether for embedded, enterprise, or CI/CD environments, `git-artifact` streamlines artifact management by making it a natural extension of your existing Git processes.

Triggering of new builds or tests are done the normal way as known from triggering your pipelines of source code - push or pull - simple..
## Quick Start

### Save money
```bash
# 1. Create a new repository on GitHub [Make sure to not initialize it with a README or .gitignore]
open https://github.com/new?name=test-git-artifact

# Waits until you are done
echo "Press Enter after creating the repository..." && read

# 2. Install git-artifact binary in your PATH
git clone https://github.com/Praqma/git-artifact.git

export PATH=$(pwd)/git-artifact:$PATH

git artifact -h

# 3. Initialize the repository locally (replace _USER_ with your GitHub username)
git artifact init --url=git@github.com:_USER_/test-git-artifact.git --path test-git-artifact && cd test-git-artifact

# 4. Add an artifact
touch artifact-1.0
git artifact add-n-push -t v1.0

# 5. Find and retrieve the latest artifact
git artifact find-latest
git artifact fetch-co-latest
```

No external tools or complex configurations are needed. `git-artifact` leverages Git's powerful version control features to manage artifacts as if they were part of your source code.

## How to use `git-artifact`

### Install git-artifact

To install `git-artifact`, simply download or clone this repository and ensure the script is available in your `PATH`. Once in your `PATH`, `git` will automatically recognize `git-artifact` as a native subcommand (`git artifact`), seamlessly extending your Git functionality.

Or quickly install `git-artifact` by running:

```bash
mkdir -p ~/.local/bin && git clone https://github.com/Praqma/git-artifact.git ~/.local/git-artifact && ln -sf ~/.local/git-artifact/git-artifact ~/.local/bin/git-artifact && git artifact -h
```

Make sure `~/.local/bin` is included in your `PATH` environment variable:

```bash
# For bash users
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc
```

```zsh
# For zsh users
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc
```

```bash
# Verify installation
git artifact -h
# Note: Use -h instead of --help (--help looks for a man page which isn't installed)
```

Now you can use `git artifact` as a regular Git command.

### Create a test repository

- Create a repo, for example on GitHub: [https://github.com/new?name=test-git-artifact]

- Use the `git artifact init` command to initialize a new repository locally for managing artifacts.

```bash
# replace `_USER_` with your GitHub username
git artifact init --url=git@github.com:_USER_/test-git-artifact.git --path test-git-artifact
```

### Add the artifact

Copy the artifacts to your path in the folder structure the "consumer" desires. There is no reason to tar or zip it. Git will handle this for optimized storage and ease of use.

```bash
cd test-git-artifact
touch artifact-1.0
# cp -rf <build-dir>/my.lib /include .
git artifact add-n-push -t v1.0
```

Voila! The artifact v1.0 is now committed, pushed _and_ importantly - the workspace is set back to the default branch of the remote repository. It is now ready to make a new artifact based on the default branch.

### Finding and getting artifacts

```bash
# You can use the `git artifact clone` command to clone a repository and set it up for artifact management. Note that you only clone and get the default branch!
git artifact clone --url=git@github.com:_USER_/test-git-artifact.git --path test-git-artifact
cd test-git-artifact
```

```bash
# Find the latest using pattern
git artifact find-latest -r 'v*.*'
```

```bash
# Download and checkout the latest
git artifact fetch-co-latest --regex 'v*.*'
```

## Notes

### Permissions needed

`git-artifact` relies on Git tags for artifact management. As a producer, you need permission to create and push tags to the remote repository. To effectively manage and clean up old artifacts, having permission to delete tags is also recommended.

You can save additional license and maintainance cost and/or coqnative load by maintaining and using an additional system for artifacts. You can handle it with your current git repository manager.
Alternatively, `git-artifact` can operate in branch mode, maintaining a `latest` branch to track the most recent artifact. This requires force-push or delete-and-push rights for the branch. The approach is similar to Docker’s `<image>:latest` tag and is useful if you want to use tracking branches outside of `git-artifact`—for example, with Git submodules or repo manifests.

### Producer of artifacts

A few remarks, aspects and thoughts when storing the artifacts
A few remarks, aspects and thoughts when storing artifacts:

- easy to append artifacts as stages evolves with more artifacts
- no need to zip before upload - just commit as the artifact should be used.
- easy to add information, environment, tools and git source sha1 in the artifact for traceability and later reproduction
- add the source code as a dependency to the artifact. It will then be easy restore the source for diff and debugging
- Easy to append artifacts as stages evolve with more artifacts
- No need to zip before upload - just commit as the artifact should be used
- Easy to add information, environment, tools and git source sha1 in the artifact for traceability and later reproduction
- Add the source code as a dependency to the artifact. It will then be easy to restore the source for diff and debugging

### Consumer of the artifacts

A few remarks, aspects and thoughts when retrieving the artifacts
A few remarks, aspects and thoughts when retrieving artifacts:

- The consumer do not need anything than standard git
- Pipelines just consumes the artifact unzip and ready to use as they were produced
- Use your favorit git dependency system like submodules(this is the correct way for submodule usage btw ), repo tool or ..
- Even a consumer can be a producer by adding further artifacts on top the consumed commit with a new commit and tag
- git understand the content in workspace and git clean does not remove artifacts in contrast to downloaded artifacts
- The consumer does not need anything other than standard git
- Pipelines just consume the artifact unzipped and ready to use as they were produced
- Use your favorite git dependency system like submodules (this is the correct way for submodule usage btw), repo tool or similar
- Even a consumer can be a producer by adding further artifacts on top of the consumed commit with a new commit and tag
- Git understands the content in workspace and git clean does not remove artifacts in contrast to downloaded artifacts

## How is it done
### How is it done

Git normally stacks the history hence you cannot delete commit in the middle of the history. `git-artifact` make a "horizontal" history - i.e the commits are not stacked on top of each other, but next to each other.
Git normally stacks the history; hence you cannot delete commits in the middle of the history. `git-artifact` makes a "horizontal" history - i.e., the commits are not stacked on top of each other, but next to each other.

The history of git-artifact workflow can basically look like this:
The history of the git-artifact workflow can basically look like this:

``` mermaid
```mermaid
%%{init: {
'gitGraph': {
'loglevel' : 'debug',
Expand Down Expand Up @@ -73,73 +180,18 @@ gitGraph:
checkout latest-2.0
commit id: "2.0/src" tag: "2.0/src"
commit id: "2.0/test" tag: "2.0/test"
```



### Prerequisites

The tool uses tags hence the producer need to tag push-rights. It is also beneficial to have tag delete-rights to clean old artifacts.

It can also run in branch mode. It can maintain a `latest` branch which needs to be force pushed or delete + push rights. The concept is similar to docker concept of `<image>/latest`. It is only important if you want to use tracking branches without using `git-artifact`. It could be in context of `submodules` or `repo manifests`.

### Installation

Download or clone this repo (`git-artifact`) and add make it available in the PATH. Given that `git-artifact` is in the PATH, then `git` can use it as an command like `git artifact`. It is now integrated with git and git is extended.

## Getting started

First you need to create a repository in your git repository manager. You can either choose to initialize the repo, but `git artifact` also have command to do that and it states that this repository is "special" repository containing git artifacts.

### Initialize the repository

Let us now initialized a repo:

```bash
git artifact init --url=<remote_url> --path my-git-artifact-path
```

### Add the artifact

The repository and path initialize above ready to use. Copy the artifacts to your path in the folder structure the "constumer" desire it. There is not reason to tar or zip it. Git will handling this for optimized storage and easiness.

```bash
cd my-git-artifact-path
cp -rf <build-dir>/my.lib /include .
git artifact add-n-push -t v1.0
```

The artifact v1.0 is now commited, pushed _and_ importantly - the workspace is set back to the default branch of the remote repository. It is now ready to make a new artifact based on the default branched

## Finding and getting artifacts

Firstly clone the git artifact repository. Note that you only clone and get the default branch

```bash
git artifact clone --url=<remote> --path my-git-artifact-path
cd my-git-artifact-path
````

### Find the latest using pattern

```bash
git artifact find-latest -r 'v*.*'
```

### Download and checkout the latest

```bash
git artifact fetch-co-latest --regex 'v*.*'
```
## Advanced

## Appending to an artifact
### Appending to an artifact

You can append to an artifact with advantage. Let say you create a library and you run a lot of tests in a later stage and the result is a test report. You can then just add that on top of the library tag.
You can append to an artifact advantageously. Let's say you create a library and you run a lot of tests at a later stage and the result is a test report. You can then just add that on top of the library tag.

- Download and checkout the artifact ( see above )
- Add a new artifact ( see above )

You should of course consider this in your naming convension. Consider something like this:
You should of course consider this in your naming convention. Consider something like this:

```bash
vX.Y.Z/release-note
Expand All @@ -148,40 +200,38 @@ vX.Y.Z/src
vX.Y.Z/lib
```

### Add the source code that was used to build the artifact

The source code in many companies and open-source projects are free to view, debug and edit. You can make it easy accessable by adding the source code as submodule and sha1 in to the artifact history. It sounds odd, but it gives the developers easy access to checkout the correct version that was used to build artifact.
#### Add the source code that was used to build the artifact

This way it actually possible to create a full block-chain of everything that was involved in producing a product.
The source code in many companies and open-source projects is free to view, debug and edit. You can make it easily accessible by adding the source code as a submodule and sha1 into the artifact history. It sounds odd, but it gives developers easy access to checkout the correct version that was used to build the artifact.

## Add information to the annotated tag

TODO: option for file or string

## Pruning / cleaning artifacts

TODO: based on count..

## Advanced
This way it is actually possible to create a full blockchain of everything that was involved in producing a product.

### LFS

`git artifact` work great out of the box without any extensions like LFS. It can though still be interesting to commit an `git-lfs` configuration to the default branch
`git artifact` works great out of the box without any extensions like LFS. It can though still be interesting to commit a `git-lfs` configuration to the default branch

- Artifact sets that can many common binary/large files from version to version will then be able to detect that it already have have this file in the LFS storage and do not have to fetch/push it again.
- You can download all tags without checkout and then you can search for meta-data in the annotated tags without suffering large data transfer and storage in order to clean.
- Artifact sets that contain many common binary/large files from version to version will then be able to detect that they already have this file in the LFS storage and do not have to fetch/push it again.
- You can download all tags without checkout and then you can search for metadata in the annotated tags without suffering large data transfer and storage in order to clean up.

### Promotions

There are genrally default two ways to you can do promotions.
Building new artifacts for the release is like a new artifact using the above patterns, which can either be a new or appended artifacts.
There are generally two default ways you can do promotions.
Building new artifacts for the release is like creating a new artifact using the above patterns, which can either be a new or appended artifact.

Promotion decision should also be seen in connection related to pruning of tag which is not valid of any interest anymore. It should be simple and easy to prune without fear of deleting tags that should not be deleted
Promotion decisions should also be seen in connection to pruning of tags which are no longer valid or of any interest. It should be simple and easy to prune without fear of deleting tags that should not be deleted.

#### Using different repository

This way is like promotion in normal artifact managemnet systems, where you promote to from one project/repository to another. You basically download the tag from the original repository and then push the tag to promotion reposity. This way you can control access and keep different URL's for candidates and releases.
This way is like promotion in normal artifact management systems, where you promote from one project/repository to another. You basically download the tag from the original repository and then push the tag to the promotion repository. This way you can control access and keep different URLs for candidates and releases.

#### Using same repository

This way requires you to create a tag using a release tag pattern. The tag can either be a new unrelated tag or it can be append on top if a release candidate tag.
This way requires you to create a tag using a release tag pattern. The tag can either be a new unrelated tag or it can be appended on top of a release candidate tag.

### Add information to the annotated tag

TODO: option for file or string

### Pruning / cleaning artifacts

TODO: based on count..
15 changes: 8 additions & 7 deletions _tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ function generate_base_repo() {
git init --bare $remote_tester_repo
git -C $remote_tester_repo symbolic-ref HEAD refs/heads/${default_branch:-main}
}
git artifact init --url="$(pwd)/$remote_tester_repo" --path $local_tester_repo -b ${default_branch:-main}
git artifact init --url="$(pwd)/$remote_tester_repo" --path $local_tester_repo -b ${default_branch:-main}
cd $local_tester_repo
touch test.txt
git artifact add-n-push -t v1.0
Expand Down Expand Up @@ -138,7 +138,7 @@ eval_testcase
test="4"
testcase_synopsis="base-repo ; clone; add-n-push with branch"
testcase_header
{
{
cd $test
generate_base_repo
git artifact clone --url=$(pwd)/$remote_tester_repo -b latest --path $clone_tester_repo
Expand All @@ -153,7 +153,7 @@ eval_testcase
test="5"
testcase_synopsis="base-repo ; clone; fetch-co-latest pattern"
testcase_header
{
{
cd $test
generate_base_repo
sleep 1
Expand Down Expand Up @@ -186,7 +186,7 @@ eval_testcase
test="5.1"
testcase_synopsis="base-repo ; clone; find-latest pattern"
testcase_header
{
{
cd $test
generate_base_repo
sleep 1
Expand All @@ -200,7 +200,7 @@ eval_testcase
test="6"
testcase_synopsis="base-repo ; clone; fetch-tags"
testcase_header
{
{
cd $test
generate_base_repo
git artifact clone --url=$(pwd)/$remote_tester_repo -b latest --path $clone_tester_repo
Expand Down Expand Up @@ -235,6 +235,7 @@ else
echo "Some tests failed. - List of logs:"
find . -name run.log -o -name nok.log | sort
echo
fi
fi

# Exit with the global exit code
exit $global_exit_code
exit $global_exit_code
Loading