Add ability to specify specific cartridges

[Jakobud]

Currently the "cartridges" option is the path to a directory containing ALL your cartridges. And as a result, dw-utils watches ALL the cartridges.

It would be really nice to be able to specify individual cartridges that you wish to watch/upload. For example I have a lot of build scripts that do various things, including creating temporary folders and files in certain subdirectories. I don't want to watch those or upload them. But since dw-utils does a blanket watch on the entire directory and ALL subdirectories there is no way to avoid it.

It would be easy enough to change the "cartridges" option to accept an array of cartridge paths to watch. But it would probably render the current usage of the option non-backwards compatible.

I would like to mess with this and put in a pull request for it if you are okay with it. But I could use some direction on if I should change the "cartridges" option functionality or specify a new option, etc...

In my opinion, it might be wise to simply be able to specify a Glob of paths to watch/upload. That would make it so it still supports specify a single path to all cartridges but would allow someone to be more specific and specify multiple paths to individual cartridges.

Thoughts?

[Jakobud]

Just looking over the code, looks like Chokidar is used to watch the files. According to that, the .watch() method accepts a string or array of paths or glob patterns. Should this already be working in dw-utils but just not documented?

https://github.com/paulmillr/chokidar

[Jakobud]

Continuing some research here. It looks like all the code treats "cartridges" as a single path. If the code was adjusted to loop through an array of paths then that would be a huge improvement.

Can you give any insight into what the Clean option is doing? How would that work if multiple cartridge paths were supported?

[MobiusHorizons]

@Jakobud, dw-utils uses the cartridges parameter for a number of different things. for the watch command, it sets up watchers using chokadir as you mentioned, but it also uses it as the base for the directory structure when it uploads zip files. (this is what happens when you see bulk uploading). dw-utils clean also uses the cartridges parameter, and currently assumes it's a string. There is already a ticket to allow multiple cartridge directories, while slightly different I think the fix for this will be the same.

I don't really have time to work on either of these right now, but I will happily accept pull requests.

[Jakobud]

Okay so, let's hypothetically say the cartridges option was changed to be an array of cartridges to watch/upload/etc... Let's say you have the following cartridges:

app_storefront
app_storefront_ui
app_custom
app_somelink
org_somethingcustom

And lets say you set the cartridges option to watch only app_custom and org_somethingcustom because those are the only cartridges you care to watch for changes. If you did this, I would imagine this would cause problems with the other functions, like Clean, right? It's hard to clean or do a bulk upload if dw-utils doesn't know about all the cartridges that exist.

So I think the problem here lies in that the cartridges option is responsible for multiple things. You can't change it to specify anything less than ALL the cartridges or else the bulk upload feature would fail to upload everything needed to the sandbox. Is that correct?

So I see 2 options here:

1. Add more options. Maybe add a watch or cartridgeWatch option that accepts an array of glob values that get watched. So the Watch functionality would use this instead of the cartridges option.

2. I see that chokidar supports some other options and perhaps a simple solution is to expose these options in dw-utils. For example there is an ignore option, where you can specify patterns for the chokidar to ignore while watching. So maybe a simple solution is to add an ignore or watchIgnore option to dw-utils that would get passed to chokidar? Or maybe a more flexible feature would be to add an option like chokidarOptions or watchOptions where you would specify any number of options that gets passed to chokidar. You could then specify the ignore option and any other options you want.

I'm leaning toward exposing a simple ignore (or watchIgnore, do you have a preference?) option that would get passed to chokidar. I think this would be a pretty easy solution toward being more specific about which cartridges you want to watch and which ones you don't and it wouldn't affect the other functions (Clean, etc) either.

What do you think?

[Jakobud]

I just finished up adding some watch ignore functionality in this branch:

Jakobud@b362a4e

Take a look and let me know what you think. Specifically it works like this:

• You can specify a single string or glob ignore pattern using the --ignore command line option. Example: dw-utils --ignore **/.git watch
• You can use the ignore option in your dw.json file where you can specify an array of strings and globs to ignore. Example:

...
"ignore": [
  "**/.git",
  "**/gulp-tmp/**/*.js"
],
...

• You can specify your ignored patterns (strings and globs) in an external file (like a .gitignore file) and specify it using either the --ignore-from option or the "ignore-from" value in your dw.json.

One things that stuck out to me is this line:

Jakobud@b362a4e#diff-761a95718c9050f2a52aa03350d58a41R228

If you get this error, this out() function seems to write over the "Connecting... Success" output in the terminal. I'm not sure how to get around that problem.

Got any thoughts? Want me to do a PR for this?

On a related note, I'm really curious as to why you haven't had any sort of ignore functionality in here already. I mean, the current dw-utils even watches and uploads your .git directory... Is that intentional? I can't imagine anyone who would be wanting to upload that into their SFCC instance...

[MobiusHorizons]

@Jakobud it only watches your .git directory if your cartridges directory is the root of your project, and for the clients I have worked with, it was a sub-directory. Ignore definitely makes sense to have, though. If you have a chance to send over a PR with what you have, I can take a look at it.

As for watching specific directories only, my preference would be to have it as a command line option, something like

dw-utils watch "glob" 

To be honest, I have never wanted to exclude any directories from my watch, but that's because every sub-directory of cartridges was a real cartridge in my setup, and if it didn't matter, probably files weren't changing in it, so watch wouldn't pick anything up anyway.

hopes that makes sense of some design decisions.

[Jakobud]

It makes sense yes but it's a hard limitation that assumes someone's cartridges are not in the root project directory. If they are then you simply can't really use dw-utils unfortunately unless you want to do stuff like upload .git files to your sandbox.

[Jakobud]

Also, the thing about watching specific directories, specifying your cartridge path does multiple things including cleaning so I don't know how that would work.

[MobiusHorizons]

I'm proposing a customization that would just specify the glob to be passed into chokadir. It would not affect the cartridge path. that combined with your ignore functionality would do what you want i think.

[Jakobud]

What if you had an array of paths to watch? This is the type of dir structure I'm looking at:

./app_storefront
./app_storefront_ui
./app_custom
./app_somelink
./org_somethingcustom
./node_modules
./gulp-scripts
./.git

etc

[MobiusHorizons]

Anyway, send a PR, and I'll take a look.

Sad to see this didn't evolved.

Another simple solution would probably be to make sure watch (and all the other functions that rely on the cartridge path) check if the folder has a ".project" file, if not, ignore it.
That will help with projects with cartridges inside the root folder.