Track dotfile not in main folder

[IamGianluca]

First of all, thank you for this tool. I've been using it for more than a year and recommend it to all my colleagues 👍

After all this time, however, it's not entirely clear to me how to track the configuration file neovim, which is located in ~/.config/nvim/init.vim.

I can add the file using the command dotfiles --add ~/.config/nvim/init.vim. This adds a init.vim file in the ~/Dotfiles folder. However, when I pull the repo in a different machine and sync, dotfiles is not copying the init.vim in the new machine. I always have to manually cp-ing it.

I read the documentation and tried a few things, but nothing seems to solve my problem. What am I missing?

[jbernard]

Thanks, that's good to hear.

First, I'm assuming you're using the pypa version 0.6.3, correct? (as opposed to master). What you describe is certainly not supposed to happen. Can you tell me the state of the repository before and after you add the file? One way to do this is with the tree command, tree -a ~/Dotfiles before and after the --add command would be very helpful.

It's possible that this is fixed in master, but that requires migrating and maybe not something you want to worry with right now. Either way, I'll get it sorted.

[IamGianluca]

Hi Jon,

Apologies for the late reply. I didn't want to mess up my environment but encoutered the issue again so I took the time to log the results of the steps you were suggesting.

Before the --add command

$ tree -a ~/Dotfiles
/home/gianluca/Dotfiles
├── .git
│   ├── branches
│   ├── config
│   ├── description
│   ├── HEAD
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── fsmonitor-watchman.sample
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   ├── pre-receive.sample
│   │   └── update.sample
│   ├── index
│   ├── info
│   │   └── exclude
│   ├── logs
│   │   ├── HEAD
│   │   └── refs
│   │       ├── heads
│   │       │   ├── freshstart
│   │       │   └── master
│   │       └── remotes
│   │           └── origin
│   │               └── HEAD
│   ├── objects
│   │   ├── info
│   │   └── pack
│   │       ├── pack-382162484ba8438adc9a3c95d5cc723bc4d3e887.idx
│   │       └── pack-382162484ba8438adc9a3c95d5cc723bc4d3e887.pack
│   ├── packed-refs
│   └── refs
│       ├── heads
│       │   ├── freshstart
│       │   └── master
│       ├── remotes
│       │   └── origin
│       │       └── HEAD
│       └── tags
├── .gitignore
├── powerline-shell.json
├── tmux.conf
└── zshrc

17 directories, 30 files

After adding my init.vim file and some other config files ― I've splitted the configuration of my neovim plugins into submodules to make it easier to maintain them. The reason why I don't add the entire ~/.config/nvim/ folder is because there are other sub-folders in it that I don't want to keep under version control.

$ dotfiles --add ~/.config/nvim/init.vim
$ tree -a ~/Dotfiles
tree -a ~/Dotfiles
/home/gianluca/Dotfiles
├── .git
│   ├── branches
│   ├── config
│   ├── description
│   ├── HEAD
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── fsmonitor-watchman.sample
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   ├── pre-receive.sample
│   │   └── update.sample
│   ├── index
│   ├── info
│   │   └── exclude
│   ├── logs
│   │   ├── HEAD
│   │   └── refs
│   │       ├── heads
│   │       │   ├── freshstart
│   │       │   └── master
│   │       └── remotes
│   │           └── origin
│   │               └── HEAD
│   ├── objects
│   │   ├── info
│   │   └── pack
│   │       ├── pack-382162484ba8438adc9a3c95d5cc723bc4d3e887.idx
│   │       └── pack-382162484ba8438adc9a3c95d5cc723bc4d3e887.pack
│   ├── packed-refs
│   └── refs
│       ├── heads
│       │   ├── freshstart
│       │   └── master
│       ├── remotes
│       │   └── origin
│       │       └── HEAD
│       └── tags
├── .gitignore
├── init.vim
├── powerline-shell.json
├── tmux.conf
└── zshrc

17 directories, 31 files
$ dotfiles --add ~/.config/nvim/configfiles/*.vim
$ tree -a ~/Dotfiles
tree -a ~/Dotfiles
/home/gianluca/Dotfiles
├── complete.vim
├── fzf.vim
├── generic.vim
├── .git
│   ├── branches
│   ├── config
│   ├── description
│   ├── HEAD
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── fsmonitor-watchman.sample
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   ├── pre-receive.sample
│   │   └── update.sample
│   ├── index
│   ├── info
│   │   └── exclude
│   ├── logs
│   │   ├── HEAD
│   │   └── refs
│   │       ├── heads
│   │       │   ├── freshstart
│   │       │   └── master
│   │       └── remotes
│   │           └── origin
│   │               └── HEAD
│   ├── objects
│   │   ├── info
│   │   └── pack
│   │       ├── pack-382162484ba8438adc9a3c95d5cc723bc4d3e887.idx
│   │       └── pack-382162484ba8438adc9a3c95d5cc723bc4d3e887.pack
│   ├── packed-refs
│   └── refs
│       ├── heads
│       │   ├── freshstart
│       │   └── master
│       ├── remotes
│       │   └── origin
│       │       └── HEAD
│       └── tags
├── .gitignore
├── grep.vim
├── init.vim
├── keys.vim
├── powerline-shell.json
├── py.vim
├── sidebar.vim
├── tmux.conf
├── visual.vim
└── zshrc

17 directories, 39 files

[IamGianluca]

After re-adding the config files for neovim ― as showed in my previous post ― and pushed the changes to the remote in GitHub, I've tried to download them and sync my work laptop. When I do dotfiles --sync. Everything gets updated apart from all the dotfiles in the folder ~/.config/nvim/configfiles/. Please note such folder did not exist in my work laptop.

I've also tried to create the folder, and sync again, but dotfiles doesn't create the symlinks as I would expect. The folder is still empty.

$ mkdir ~/.config/nvim/configfiles/
$  dotfiles --sync
$ ls ~/.config/nvim/configfiles

[jbernard]

Ok, I see. And just to be sure, what version are you using? Did you install from pypa or are you running code from the git repository?

[IamGianluca]

I've installed dotfiles via pip.

[jbernard]

Ok, definitely a bug. In this version (0.6.4), you’re only allowed to add top-level dotfiles (files that are directly in your home directory and not nested in other directories). This kinda sucked, so packages were implemented to allow things in ~/.config to be added individually. The workaround is as follows:

Create a file ~/.dotfilesrc and in that file put:

[dotfiles]
packages = [‘config’]

This could also be stored in the repository itself in ~/Dotfiles/.dotfilesrc
And now execute:

dotifles —add ~/.config/nvim

And that will correctly store your neovim configuration in the repository.

This is still pretty limited and was a major reason I’ve been hacking on the code in master. In the next version there is no limit on nested directories, no packages or any other configuration - everything works much more smoothly. I’m nearly done with the pathlib migration and I hope to have something to test up soon.

[IamGianluca]

Thanks @jbernard

I've followed your instructions with the only exception of adding the files using the command dotfiles -add ~/.config/nvim/*.vim since I have also other folders in the ~/.config/nvim/ folder which I don't want to keep under version control.

The issue I have is that the changes are not propagated to new systems. In a new laptop I can pull the changes from GitHub and run dotfiles --sync -f but the only symlinks created are those in the ~/ directory. The symlinks for files in the ~/.config/nvim/ folder are not created in my case.

I'm still running dotfiles==0.6.4 from pypa.

[jbernard]

Alright, I’ve found some motivation and pushed a bunch of commits. What’s in master now should be much closer to what you’re looking for (it certainly is for me). If you’re willing to help me test this, I could use this to construct the installation and usage documentation. Is that something you might be interested in helping with (testing the new version)? The final result will be the 1.0 version, but there are some distinct differences from 0.6.4 in interface and behavior, so it will likely require a fresh repository. Let me know if you’re interested.

[IamGianluca]

Hi Jon,

I'm definitely interested in helping you with this. I should be able to test what in the master branch by end of day tomorrow at the latest.

Thank you again for your hard work, dotfiles saves me so much time when trying to keep in sync my dotfiles in four different machines every day!

[jbernard]

That’s great, thank you! Please don’t feel obligated, give it a try when you can and let me know how it goes, what’s working, what sucks, etc. If you never report back that’s okay too, I known we’re all busy. I’ll be writing documentation for installation and usage this week hopefully, let me know if you have any questions about those things.

In general, there are a few things to know about the current version:

• arbitrary nesting levels are supported, which means only the files themselves are stored in the repository at whatever level of nesting they’re found. This makes the whole packages feature completely go away. For example, if you add ~/.config/nvim/init.vim, this file get’s linked and the directory structure is mirrored in the repository. So some files in a directory can be managed while others are either unmanaged or members of a different repository.

• multiple repositories are supported. You can use the -r flag (see dotfiles -h) to define repository locations (or use DOTFILES_REPOS environment variable). It’s a colon-separated string, so you can define and use multiple repositories by placing a : between them. The status command will show you everything, the other commands will ask you to specify one repository if multiple a given. This will improve soon, if a particular file is a member of exactly one of the available repositories, it can be assumed that that’s the one you meant (but this isn’t coded yet).

• Support for symlink or copies is nearly done. When you add or enable a dotfile, you can specify —copy and the file will be copied instead of symlinked. Symlink is the default. I still need to commit this, but status support is there. For status of a copied file, dotfiles will compare the contents of the two files and tell you in the output if the contents are different (a conflict).

• External symlinks are supported, no more external configuration, it does the right thing. So if you you add ~/.xsession-errors and that file is symlinked to /dev/null, that’s fine, dotfiles will handle that without needing to know any more.

• There is no configuration file. I tried to either implement the features transparently or remove them to reduce the complexity. Generally there are less tunables, but dotfiles is smarter where possible.

• The UI has changed, I’m using click now, so the command line is more similar to git-style. See dotfiles -h. There are 5 commands: status, add, remove, enable, and disable. Each one of them has a few specific flags, which you can see by typing dotfiles add -h, for example. Global commands should be given before the action, so something like this would define a repository and ask status to show everything: dotfiles -r ~/my-repo status -a.

• If you want to perform an action on all files within a certain sub directory, you can use that directory as an argument and dotfiles will expand it internally. So, dotfiles add ~/.ssh will add all files below the .ssh directory. And dotfiles enable ~/.config/nvim will create links for all neovim configuration files. I’m working on a few corner cases here, so there may be some bugs at the moment.

• Every action accepts a debug flag -d, —debug that will show you want commands dotifiles would execute without actually executing anything. This is helpful to see what’s going to happen and identify logic flaws if something doesn’t look right. For example, dotfiles add -d ~/.gitconfig will print commands without making any changes to the file system.

• The tests are really broken right now from the refactoring, I need to fix them, don’t be too alarmed by pytest’s complaints.

You can install this in one of three ways:

1. Install using pip into a virtual env

   • create a virtual environment
   • pip install git+https://github.com/jbernard/dotfiles

2. install using pip from a local checkout:

   • create a virtual environment
   • cd dotfiles
   • pip install -e .

3. Run from a local checkout without a venv:

   • clone the repository
   • git submodule update —init
   • execute bin/dotfiles (it will look for the click submodule)

That should be enough to get you started. Click technically support shell completions for bash and zsh, I need to test and document that. The status command has colors if you like colors (disabled by default). Give it a spin and let me know. Thanks again!

[jbernard]

Oh, I should also mention that all of my changes lately have been using Python 3 (3.6). If you’re using version 2 there may be something that slipped through, let me know if that’s the case. Once if fix the tests I should have coverage for both versions.