#60: Composing with outdated cache and #51: Simplify package state pi…

README.md

@@ -1,4 +1,5 @@
 ## Composition Tool Specification
+
 The composition tool is a command-line tool that helps developers manage
 dependencies for their projects. It allows developers to specify the dependencies for
 a project in a "plasma-compose.yaml" file, and then fetches and installs those dependencies
@@ -8,28 +9,39 @@ The tool works by recursively fetching and processing the "plasma-compose.yaml"
 and its dependencies, and then merging the resulting filesystems into a single filesystem.
 
 ### CLI
+
 The composition tool is invoked from the command line with the following syntax:
 launchr compose [options]
 Where options are:
+
 * -w, --working-dir : The directory where temporary files should be stored during the
   composition process. Default is the .compose/packages
 * -s, --skip-not-versioned : Skip not versioned files from source directory (git only)
-* --conflicts-verbosity: Log files conflicts in format "[curent-package] - path to file > Selectef from [domain, other package or current-package]"
+* --conflicts-verbosity: Log files conflicts in format "[current-package] - path to file > Selected
+  from [domain, other package or current-package]"
+* --interactive: Interactive mode allows to submit user credentials during action (default: true)
 
 Example usage - `launchr compose -w=./folder/something -s=1 or -s=true --conflicts-verbosity`
 
-It's important to note that: if same file is present locally and also brought by a package, default strategy is that local file will be taken and package file ignored. [Different strategies](https://github.com/launchrctl/compose/blob/main/example/compose.example.yaml#L18-L35) can be difined to customize this behavior to your needs.
-
+It's important to note that: if same file is present locally and also brought by a package, default strategy is that
+local file will be taken and package file
+ignored. [Different strategies](https://github.com/launchrctl/compose/blob/main/example/compose.example.yaml#L18-L35)
+can be difined to customize this behavior to your needs.
 
 ### `plasma-compose.yaml` File Format
-The "plasma-compose.yaml" file is a text file that specifies the dependencies for a package, along with any necessary metadata and sources for those dependencies.
+
+The "plasma-compose.yaml" file is a text file that specifies the dependencies for a package, along with any necessary
+metadata and sources for those dependencies.
 The file format includes the following elements:
+
 - name: The name of the package.
 - version: The version number of the package.
-- source: The source for the package, including the type of source (Git, HTTP), URL or file path, merge strategy and other metadata.
+- source: The source for the package, including the type of source (Git, HTTP), URL or file path, merge strategy and
+  other metadata.
 - dependencies: A list of required dependencies.
 
 List of strategies:
+
 - overwrite-local-file
 - remove-extra-local-files
 - ignore-extra-package-files
@@ -38,40 +50,43 @@ List of strategies:
 Example:
 
 ```yaml
-name: myproject
-version: 1.0.0
+name: example
 dependencies:
-- name: compose-example
-  source:
-    type: git
-    ref: master
-    tag: 0.0.1
-    url: https://github.com/example/compose-example.git
-    strategy:
-      - name: remove-extra-local-files
-        path: 
-          - path/to/remove-extra-local-files
-      - name: ignore-extra-package-files
-        path:
-          - library/inventories/platform_nodes/configuration/dev.yaml
-          - library/inventories/platform_nodes/configuration/prod.yaml
-          - library/inventories/platform_nodes/configuration/whatever.yaml
+  - name: compose-example
+    source:
+      type: git
+      ref: master # branch or tag name
+      url: https://github.com/example/compose-example.git
+      strategy:
+        - name: remove-extra-local-files
+          path:
+            - path/to/remove-extra-local-files
+        - name: ignore-extra-package-files
+          path:
+            - library/inventories/platform_nodes/configuration/dev.yaml
+            - library/inventories/platform_nodes/configuration/prod.yaml
+            - library/inventories/platform_nodes/configuration/whatever.yaml
 ```
 
-
 ### Fetching and Installing Dependencies
-The composition tool fetches and installs dependencies for a package by recursively processing the "plasma-compose.yaml" files for each package and its dependencies. The tool follows these general steps:
 
-1. Fetch the package source code from the specified source location.
-2. Extract the package contents to a temporary directory.
-3. Process the "plasma-compose.yaml" file for the package, fetching and installing any necessary dependencies recursively.
-4. Merge the package filesystem into the final platform filesystem.
-5. Repeat steps 1-4 for each package and its dependencies.
+The composition tool fetches and installs dependencies for a package by recursively processing the "plasma-compose.yaml"
+files for each package and its dependencies. The tool follows these general steps:
+
+1. Check if package exists locally and is up-to-date. If it's not, remove it from packages dir and proceed to next step.
+2. Fetch the package from the specified location.
+3. Extract the package contents to a packages directory.
+4. Process the "plasma-compose.yaml" file for the package, fetching and installing any necessary dependencies
+   recursively.
+5. Merge the package filesystem into the final platform filesystem.
+6. Repeat steps 1-5 for each package and its dependencies.
 
 During this process, the composition tool keeps track of the dependencies for each package.
 
 ### Plasma-compose commands
+
 it's possible to manipulate plasma-compose.yaml file using commands:
+
 - plasmactl compose:add
 - plasmactl compose:update
 - plasmactl compose:delete
@@ -84,17 +99,14 @@ For `compose:delete` it's possible to pass list of packaged to delete.
 
 In other cases, user will be prompted to CLI form to fill necessary data of packages.
 
-Example of usage
-
+Examples of usage
 
 ```
 launchr compose:add --url some-url --type http
 launchr compose:add --package package-name --url some-url --ref v1.0.0
 launchr compose:update --package package-name --url some-url --ref v1.0.0
 
 launchr compose:add --package package-name --url some-url --ref v1.0.0 --strategy overwrite-local-file --strategy-path "path1|path2"
-launchr compose:add --package package-name --url some-url --ref v1.0.0 --strategy overwrite-local-file,remove-extra-local-files --strategy-path "path1|path2,path3|path4"
+launchr compose:add --package package-name --url some-url --ref branch --strategy overwrite-local-file,remove-extra-local-files --strategy-path "path1|path2,path3|path4"
 launchr compose:add --package package-name --url some-url --ref v1.0.0 --strategy overwrite-local-file --strategy-path "path1|path2" --strategy remove-extra-local-files --strategy-path "path3|path4"
-
-
 ```

compose/compose.go

@@ -46,6 +46,12 @@ func CreateComposer(pwd string, opts ComposerOptions, k keyring.Keyring) (*Compo
 		return nil, err
 	}
 
+	for _, dep := range config.Dependencies {
+		if dep.Source.Tag != "" {
+			launchr.Term().Warning().Printfln("found deprecated field `tag` in `%s` dependency. Use `ref` field for tags or branches.", dep.Name)
+		}
+	}
+
 	return &Composer{pwd, &opts, config, k}, nil
 }
 

compose/downloadManager.go

@@ -8,17 +8,14 @@ import (
 const (
 	// GitType is const for GIT source type download.
 	GitType = "git"
-	// SourceReferenceTag represents git tag source.
-	SourceReferenceTag = "tag"
-	// SourceReferenceBranch represents git branch source.
-	SourceReferenceBranch = "ref"
 	// HTTPType is const for http source type download.
 	HTTPType = "http"
 )
 
 // Downloader interface
 type Downloader interface {
-	Download(pkg *Package, targetDir string, kw *keyringWrapper) error
+	Download(pkg *Package, targetDir string) error
+	EnsureLatest(pkg *Package, downloadPath string) (bool, error)
 }
 
 // DownloadManager struct, provides methods to fetch packages
@@ -35,14 +32,14 @@ func CreateDownloadManager(keyring *keyringWrapper) DownloadManager {
 	return DownloadManager{kw: keyring}
 }
 
-func getDownloaderForPackage(downloadType string) Downloader {
+func getDownloaderForPackage(downloadType string, kw *keyringWrapper) Downloader {
 	switch {
 	case downloadType == GitType:
-		return newGit()
+		return newGit(kw)
 	case downloadType == HTTPType:
-		return newHTTP()
+		return newHTTP(kw)
 	default:
-		return newGit()
+		return newGit(kw)
 	}
 }
 
@@ -85,16 +82,13 @@ func (m DownloadManager) recursiveDownload(yc *YamlCompose, kw *keyringWrapper,
 
 		packagePath := filepath.Join(targetDir, pkg.GetName(), pkg.GetTarget())
 
-		// Skip package download if it exists in packages dir.
-		if _, err := os.Stat(packagePath); os.IsNotExist(err) {
-			err = downloadPackage(pkg, targetDir, kw)
-			if err != nil {
-				return packages, err
-			}
+		err := downloadPackage(pkg, targetDir, kw)
+		if err != nil {
+			return packages, err
 		}
 
 		// If package has plasma-compose.yaml, proceed with it
-		if _, err := os.Stat(filepath.Join(packagePath, composeFile)); !os.IsNotExist(err) {
+		if _, err = os.Stat(filepath.Join(packagePath, composeFile)); !os.IsNotExist(err) {
 			cfg, err := Lookup(os.DirFS(packagePath))
 			if err == nil {
 				packages, err = m.recursiveDownload(cfg, kw, packages, pkg, targetDir)
@@ -111,20 +105,30 @@ func (m DownloadManager) recursiveDownload(yc *YamlCompose, kw *keyringWrapper,
 }
 
 func downloadPackage(pkg *Package, targetDir string, kw *keyringWrapper) error {
-	downloader := getDownloaderForPackage(pkg.GetType())
+	downloader := getDownloaderForPackage(pkg.GetType(), kw)
 	packagePath := filepath.Join(targetDir, pkg.GetName())
 	downloadPath := filepath.Join(packagePath, pkg.GetTarget())
 
-	if _, err := os.Stat(downloadPath); !os.IsNotExist(err) {
-		// Skip package download if folder exists in packages dir.
+	isLatest, err := downloader.EnsureLatest(pkg, downloadPath)
+	if err != nil {
+		return err
+	}
+
+	if isLatest {
 		return nil
 	}
 
+	// Ensure old package doesn't exist in case of update.
+	err = os.RemoveAll(downloadPath)
+	if err != nil {
+		return err
+	}
+
 	// temporary
 	if dtype := pkg.GetType(); dtype == HTTPType {
 		downloadPath = packagePath
 	}
 
-	err := downloader.Download(pkg, downloadPath, kw)
+	err = downloader.Download(pkg, downloadPath)
 	return err
 }

compose/forms.go

@@ -315,7 +315,6 @@ func processStrategiesForm(dependency *Dependency) error {
 }
 
 func preparePackageForm(dependency *Dependency, config *YamlCompose, isAdd bool) *huh.Form {
-	var refType string
 	uniqueLimit := 1
 	if isAdd {
 		uniqueLimit = 0
@@ -376,27 +375,11 @@ func preparePackageForm(dependency *Dependency, config *YamlCompose, isAdd bool)
 				}),
 		),
 
-		huh.NewGroup(
-			huh.NewSelect[string]().
-				Title("- Select source reference").
-				Options(
-					huh.NewOption("Tag", SourceReferenceTag).Selected(true),
-					huh.NewOption("Branch", SourceReferenceBranch),
-				).
-				Value(&refType),
-		).WithHideFunc(func() bool { return dependency.Source.Type != GitType }),
-
-		huh.NewGroup(
-			huh.NewInput().
-				Title("- Enter Tag").
-				Value(&dependency.Source.Tag),
-		).WithHideFunc(func() bool { return dependency.Source.Type != GitType || refType != SourceReferenceTag }),
-
 		huh.NewGroup(
 			huh.NewInput().
 				Title("- Enter Ref").
 				Value(&dependency.Source.Ref),
-		).WithHideFunc(func() bool { return dependency.Source.Type != GitType || refType != SourceReferenceBranch }),
+		).WithHideFunc(func() bool { return dependency.Source.Type != GitType }),
 	)
 }
 
@@ -431,5 +414,4 @@ func sanitizeDependency(dependency *Dependency) {
 	dependency.Name = strings.TrimSpace(dependency.Name)
 	dependency.Source.URL = strings.TrimSpace(dependency.Source.URL)
 	dependency.Source.Ref = strings.TrimSpace(dependency.Source.Ref)
-	dependency.Source.Tag = strings.TrimSpace(dependency.Source.Tag)
 }