Compile fails reinstalling packages (in some instances)

[nuc1eon]

Certain packages fail to build if they are already installed under /Programs/*.

It can be reproduced by installing LibBPF 0.2 and trying to reinstall it using Compile LibBPF 0.2. In the presented scenario, the installation will fail with the following error:
real_install: cannot change permissions of ‘/usr/include/bpf’: No such file or directory.

Doing a RemoveProgram LibBPF 0.2 prior to calling Compile LibBPF 0.2 will make it work.

Unless a user already knows this, this can be a very frustrating experience.

[nuc1eon]

In other cases Compile fails to update an existing program, for example Kmod 27 -> Kmod 32.

The solution is to run eg DisableProgram Kmod prior to installation.

Updating Vim shows a similar behaviour as @sage-etcher pointed out.

[nuc1eon]

Upon trying to "reinstall" LibXdmcp 1.1.5, I have accidentally run rm -rf /Programs/LibXdmcp, however despite that I was getting:

 /System/Index/bin/mkdir -p '/usr/share/doc/libXdmcp'
 /System/Index/bin/mkdir -p '/usr/share/doc/libXdmcp'
/System/Index/bin/mkdir: cannot create directory ‘/usr/share/doc/libXdmcp’: File exists
/System/Index/bin/mkdir: cannot create directory ‘/usr/share/doc/libXdmcp’: File exists
make[2]: *** [Makefile:439: install-dist_shelfDATA] Error 1

Obviously I could not run DisableProgram LibXdmcp 1.1.5 as I usually do, so I ran find /System/Index | RemoveBroken - and that fixed the install step.

So the solution might be to RemoveBroken symlinks before installing the files?

UPDATE:

Indeed, I was able to fix the "File exists" issue by adding:

echo "find /System/Index | RemoveBroken > /dev/null" >>"${sandbox_mp}/.wrapper"

between lines 209 and 210 at the bottom:

Compile/bin/UnionSandbox

Lines 186 to 211 in d94f386

	# Create wrapper
	newpwd="$PWD"
	Is_Entry_Set "directory" && newpwd=`Entry "directory"`
	echo "#!/bin/bash" >"${sandbox_mp}/.wrapper"
	echo "cd $newpwd" >>"${sandbox_mp}/.wrapper"
	Args_To_Array command >> "${sandbox_mp}/.wrapper"
	if [ "$unionbackend" != "unionfs" ]
	then
	# delete entries in sandbox if they are legacy symlinks
	echo "[ -L $goboDevices ] && rm $goboDevices" >> "${sandbox_mp}/.wrapper"
	echo "[ -L $goboStatus ] && rm $goboStatus" >> "${sandbox_mp}/.wrapper"
	if grep -q devtmpfs "$goboStatus/filesystems"
	then devfs=devtmpfs
	else devfs=tmpfs
	fi
	echo "mkdir -p $goboDevices; mount -t $devfs none $goboDevices" >> "${sandbox_mp}/.wrapper"
	echo "mkdir -p $goboStatus; mount -t proc none $goboStatus" >> "${sandbox_mp}/.wrapper"
	echo "[ -e \"${goboLibraries}/udev/devices/\" ] && cp -aR \"${goboLibraries}/udev/devices/\"* $goboDevices" >> "${sandbox_mp}/.wrapper"
	# We check if udevadm is also available at runtime. This is needed when building Udev itself.
	if Executable_Exists_In_Path "udevadm"
	then echo "which udevadm >& /dev/null && udevadm trigger && udevadm settle --timeout=10" >> "${sandbox_mp}/.wrapper"
	else echo "which udevtrigger >& /dev/null && udevtrigger && udevsettle" >> "${sandbox_mp}/.wrapper"
	fi
	fi
	echo "\"\${command[@]}\"" >>"${sandbox_mp}/.wrapper"

/.wrapper is a wrapper script inside the union sandbox, which then runs a variation of make install.

Also, I found a useful debugging shell that can be activated via uncommenting:

Compile/bin/UnionSandbox

Lines 213 to 216 in d94f386

	# Useful for debugging:
	#echo "res=\$?" >>"${sandbox_mp}/.wrapper"
	#echo "echo DEBUG SHELL; bash" >>"${sandbox_mp}/.wrapper"
	#echo "exit \$res" >>"${sandbox_mp}/.wrapper"

This puts one straight into the sandbox, a chrooted shell environment.

UPDATE 2:

I wanted to investigate why, for instance upgrading LibXdmcp 1.1.4 -> LibXdmcp 1.1.5 did not cause any issue, while reinstalling the same version eg LibXdmcp 1.1.4 or LibXdmcp 1.1.5 "onto itself" would fail with:

/System/Index/bin/mkdir: cannot create directory ‘/usr/share/doc/libXdmcp’: File exists

It seems that when a we do a reinstallation of let's say v1.1.5, the directory /usr/share/doc/libXdmcp is a broken link to /Programs/LibXdmcp/1.1.5/share/doc/libXdmcp. Therefore the folder is inaccessable.

On the other hand when upgrading from LibXdmcp 1.1.4 to 1.1.5, the directory /usr/share/doc/libXdmcp exists and (!) points to /Programs/LibXdmcp/1.1.4/share/doc/libXdmcp. So the files from v1.1.5 basically get installed into an existing dir which was set up by v1.1.4 (!).

On a "clean" install of version 1.1.4 or 1.1.5 (where no previous version exists), the install script of said program simply freshly creates the directory.

So one could say that some install scripts may get "confused" by broken symlinks (or broken directory symlinks... at least in our showcases).

I suppose a similar thing happens in the case of LibBPF (mentioned in the first post).

The same thing happens for the Program File 4.45, which too, spits out an error like: ‘/usr/share/misc’: File exists when installed "onto" itself.

Conclusion

For the programs LibXdmcp, File, LibBPF and probably many others, cleaning broken symlinks before install (my above mentioned fix) is a working antidote!!

However there are still corner cases which will continue to fail, even with this fix, for instance VIM 9.1.0224, or Kmod < 33 (it actually got fixed in Kmod 33 see kmod-project/kmod@c5054b2).

These above mentioned programs, simply have install scripts that do not allow "overwriting" existing files (installing into a "dirty" dir). This is either by sloppy packaging or maybe intended by the package author.

These programs will fail to install if any version (current or old) exists symlinked inside /System/Index.

Since we cannot control how a packages' install script is written we have two options for these type of packages:

1. Patch the install script to allow overwriting (like we did for VIM 9.1.0224, see patch, Vim 9.1.0224 Recipes#221)
2. Unlink (or "DisableProgram)" any old versions of the same package inside the UnionSandbox, so that we "always" have a "clean" install target.

Caveat of method 2:

In some corner cases disabling old versions of a certain package during the install phase, could (granted, it's an educated guess) lead to issues, for instance our Compile script might depend on it.

I think for that reason we should, for now, withhold implementing option 2) and possibly opt for option 1).

In order to implement option 2) (which seems like the cleanest approach), we would have to ensure that our own in-house tools, are self-contained, in the sense that they would not depend on programs inside /System/Index (at least during install).

Since currently this is not the case, we do actually face difficulties when updating certain programs or tools which Compile itself depends on.. for instance updating Python, Curl can be challenging or lead to broken states.

Even above mentioned File actually shows an (probably harmless but still) error while being reinstalled, because a library of 'itself' is missing that Compile makes use of during install (EDIT: just tried to re-produce to get the err msg, but couldn't lol. But the point is still there I guess)

I think our system tools deserve to be a bit more solid than that.

---

@sage-etcher fyi, you've stumbled upon that error with VIM so you might be interested in my findings..

[nuc1eon]

The rabit hole just got deeper...

I have now reached an understanding how these broken symlinks under /System/Index/ come to be during build using Compile.

The program ACL is a good example:

Compile runs a Pre_Installation_Preparation script before the install step, which moves stuff from /Programs/ACL/2.3.2/{bin,lib,share,...} -> /Programs/ACL/2.3.2-safelinking/{...}. Since that move breaks symlinks in /System/Index, the Pre_Installation_Preparation handles that via line 163 - it converts all symlinks to *-safelinking.

This gets reverted later within the Post_Installation_Clean_Up function.

However the problem is Runner. If Runner is used, apparently moving /Programs/ACL/2.3.2/{bin,lib,share,...} -> /Programs/ACL/2.3.2-safelinking/{...} works correctly, however the amendments to the symlinks in /System/Index do not have any effect. I checked that the code is executing correctly, but it just doesn't affect the real system...

This is a huge issue, as for instance the program ACL is a dependency of sed, so Compile will brick upon re-installing ACL.

Unfortunately I don't know how to fix this easily. The way the code is written I do not know how to just "exit" Runner temporarily. I suppose it would be better if the Runner operation ran in a separate process, because exit 0 will exit the whole Compile process. But that would require a bigger code re-arrangement within bin/Compile. :/

If I understand correctly Compile simply respawns with Runner if available and then performs all the steps... build, install,... from within that space. I do not see a way to "break out" of it easily...

@lucasvr I'd really appreciate your thoughts on how to approach this if you have any time to spare.

[nuc1eon]

I have added commit 2e5ab28 that mitigates 90% of re-install issues by removing broken symlinks on /System/Index before installing.

This has to be fixed properly by running Pre_Installation_Preparation outside of Runner.

Unless I am mistaken, this would require a bigger restructuring of bin/Compile. Because atm Compile simply respawns with Runner as a whole, instead of selectively spawning Runner upon build, install, etc...
Pre_Installation_Preparation is supposed to be run right between the build and install phase (and outside of Runner).

2e5ab28 will mitigate the cases where install scripts get confused by "broken folder symlinks".

However it will not fix re-installation of programs like ACL (programs that are essential for the system to work propely), because these programs will be temporarily 'unsymlinked' during the install phase, leading to a broken system state.