Conditionally disable command line length work-around in `Makefile.in`

[jaganmn]

The work-around in question is here:

flint/Makefile.in

Lines 411 to 430 in 74907c3

	ifneq ($(SHARED), 0)
	shared: $(FLINT_DIR)/$(FLINT_LIB_FULL)
	# The following is to avoid reaching the maximum length of command line
	# arguments, mainly present on MinGW.
	define xxx_merged_lobj_rule
	$(BUILD_DIR)/$(1)_merged.lo: $($(1)_LOBJS) | $(BUILD_DIR)
	@$(LD) -r $($(1)_LOBJS) -o $(BUILD_DIR)/$(1)_merged.lo
	endef
	$(foreach dir, $(DIRS), $(eval $(call xxx_merged_lobj_rule,$(dir))))
	MERGED_LOBJS:=$(foreach dir, $(DIRS),$(BUILD_DIR)/$(dir)_merged.lo)
	$(FLINT_DIR)/$(FLINT_LIB_FULL): $(MERGED_LOBJS)
	@echo "Building $(FLINT_LIB_FULL)"
	@$(CC) $(CFLAGS) -shared $(EXTRA_SHARED_FLAGS) $(MERGED_LOBJS) -o $(FLINT_LIB_FULL) $(LDFLAGS) $(LIBS)
	@$(RM_F) $(FLINT_LIB)
	@$(RM_F) $(FLINT_LIB_MAJOR)
	@$(LN_S) $(FLINT_LIB_FULL) $(FLINT_LIB)
	@$(LN_S) $(FLINT_LIB_FULL) $(FLINT_LIB_MAJOR)
	endif

Unfortunately, not all linkers support the -r option. Notably the MinGW backend for lld, the LLVM linker, does not support it, preventing cross-compiling of FLINT under Unix for aarch64 Windows: https://github.com/llvm/llvm-project/blob/main/lld/MinGW/Options.td

Can this work-around be used only on Windows and systems where it is known to be necessary?

[albinahlback]

They must have something to merge files, no?

[jaganmn]

IIUC, on Windows one would use lib.exe to build object libraries from sets of object files, then use link.exe to link the object libraries. Since the MinGW backend of lld was designed to emulate link.exe, there need not be a simple replacement for ld -r.

Disabling the work-around on systems without known command line length limitations seems safest/easiest.

And, to be clear, we are just talking about one of the lld backends. The ELF backend fully supports -r.

[albinahlback]

I'm sure I am misunderstanding something; you would like to disable merging of files for MinGW, that one system that actually requires it in order compile FLINT?

[jaganmn]

No. We are running make under a Linux without limitations on command line length. However, because we are cross-compiling for Windows (building on Linux binaries to be distributed on Windows), we are using as our linker the MinGW backend of lld. This linker does not support -r, so we are hitting a linker error under Linux when building FLINT for Windows.

The proposal is to "merge" only if the system running make is known to have limitations on command line length.

[albinahlback]

I have thought about this for a little bit, and I do not think this is an issue about FLINT. I believe this is a pretty standard feature of linkers, and so I would first like to see this discussed at LLVM before trying to solve it here.

However, I am still opening to merging relevant PRs that may resolve this issue.

[jaganmn]

OK. In the mean time, we will work around this with a patch against the latest release (3.1.3-p1). I've just tested the following patch under each of x86_64-pc-linux-gnu and aarch64-apple-darwin23.6.0:

diff -ruN flint-3.1.3-p1/Makefile.in flint-3.1.3-p1-patched/Makefile.in
--- flint-3.1.3-p1/Makefile.in	2024-05-24 08:52:40
+++ flint-3.1.3-p1-patched/Makefile.in	2024-10-20 23:32:18
@@ -416,6 +416,10 @@
 ifneq ($(SHARED), 0)
 shared: $(FLINT_DIR)/$(FLINT_LIB_FULL)
 
+ifneq ($(strip $(filter gnu% linux-gnu%,@FLINT_BUILD_OS@)),)
+# No command line length limitations under build_os=gnu*|linux-gnu*
+MERGED_LOBJS := $(foreach dir,$(DIRS),$($(dir)_LOBJS))
+else
 # The following is to avoid reaching the maximum length of command line
 # arguments, mainly present on MinGW.
 define xxx_merged_lobj_rule
@@ -424,6 +428,7 @@
 endef
 $(foreach dir, $(DIRS), $(eval $(call xxx_merged_lobj_rule,$(dir))))
 MERGED_LOBJS:=$(foreach dir, $(DIRS),$(BUILD_DIR)/$(dir)_merged.lo)
+endif
 
 $(FLINT_DIR)/$(FLINT_LIB_FULL): $(MERGED_LOBJS)
 	@echo "Building $(FLINT_LIB_FULL)"
@@ -437,6 +442,10 @@
 ifneq ($(STATIC), 0)
 static: $(FLINT_DIR)/$(FLINT_LIB_STATIC)
 
+ifneq ($(strip $(filter gnu% linux-gnu%,@FLINT_BUILD_OS@)),)
+# No command line length limitations under build_os=gnu*|linux-gnu*
+MERGED_OBJS := $(foreach dir,$(DIRS),$($(dir)_OBJS))
+else
 # The following is to avoid reaching the maximum length of command line
 # arguments, mainly present on MinGW.
 define xxx_merged_obj_rule
@@ -445,6 +454,7 @@
 endef
 $(foreach dir, $(DIRS), $(eval $(call xxx_merged_obj_rule,$(dir))))
 MERGED_OBJS:=$(foreach dir, $(DIRS),$(BUILD_DIR)/$(dir)_merged.o)
+endif
 
 $(FLINT_DIR)/$(FLINT_LIB_STATIC): $(MERGED_OBJS)
 	@echo "Building $(FLINT_LIB_STATIC)"
diff -ruN flint-3.1.3-p1/configure.ac flint-3.1.3-p1-patched/configure.ac
--- flint-3.1.3-p1/configure.ac	2024-05-24 08:52:40
+++ flint-3.1.3-p1-patched/configure.ac	2024-10-20 23:25:02
@@ -129,7 +129,11 @@
 
 dnl Get system triplet
 dnl NOTE: This is already invoked from LT_INIT
+dnl AC_CANONICAL_BUILD
 dnl AC_CANONICAL_HOST
+
+FLINT_BUILD_OS="${build_os}"
+AC_SUBST(FLINT_BUILD_OS)
 
 ################################################################################
 # configure headers

It differs from my proposal by "merging" except under GNU/Linux, rather than only under MinGW.

[kalibera]

For reference, the error output from the linker is simply

lld: error: unknown argument: -r

Asking for -r to be supported just based on problems with command line length limits might be a hard sell. Perhaps instead of merging object files, you could iteratively keep adding them to a static library, and later use that one for linking?

[oscarbenjamin]

I'm hitting up against this in flintlib/python-flint#360. There I am trying to build FLINT natively on Windows on arm64 using msys2. I'm no expert on these things but I think I am trying to use the same toolchain as @jaganmn except running it natively rather than cross-compiling.

With FLINT as it stands I get

lld: error: unknown argument: -r

The patch above won't help because it checks the build OS but I can change it to this instead:

diff --git a/Makefile.in b/Makefile.in
index 0f25aa2b5..937fffa10 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -416,14 +416,7 @@ endif
 ifneq ($(SHARED), 0)
 shared: $(FLINT_DIR)/$(FLINT_LIB_FULL)
 
-# The following is to avoid reaching the maximum length of command line
-# arguments, mainly present on MinGW.
-define xxx_merged_lobj_rule
-$(BUILD_DIR)/$(1)_merged.lo: $($(1)_LOBJS) | $(BUILD_DIR)
-       @$(LD) -r $($(1)_LOBJS) -o $(BUILD_DIR)/$(1)_merged.lo
-endef
-$(foreach dir, $(DIRS), $(eval $(call xxx_merged_lobj_rule,$(dir))))
-MERGED_LOBJS:=$(foreach dir, $(DIRS),$(BUILD_DIR)/$(dir)_merged.lo)
+MERGED_LOBJS:=$(foreach dir,$(DIRS),$($(dir)_LOBJS))
 
 $(FLINT_DIR)/$(FLINT_LIB_FULL): $(MERGED_LOBJS)
        @echo "Building $(FLINT_LIB_FULL)"
@@ -437,14 +430,7 @@ endif
 ifneq ($(STATIC), 0)
 static: $(FLINT_DIR)/$(FLINT_LIB_STATIC)
 
-# The following is to avoid reaching the maximum length of command line
-# arguments, mainly present on MinGW.
-define xxx_merged_obj_rule
-$(BUILD_DIR)/$(1)_merged.o: $($(1)_OBJS) | $(BUILD_DIR)
-       @$(LD) -r $($(1)_OBJS) -o $(BUILD_DIR)/$(1)_merged.o
-endef
-$(foreach dir, $(DIRS), $(eval $(call xxx_merged_obj_rule,$(dir))))
-MERGED_OBJS:=$(foreach dir, $(DIRS),$(BUILD_DIR)/$(dir)_merged.o)
+MERGED_OBJS:=$(foreach dir,$(DIRS),$($(dir)_OBJS))
 
 $(FLINT_DIR)/$(FLINT_LIB_STATIC): $(MERGED_OBJS)
        @echo "Building $(FLINT_LIB_STATIC)"

That gets past the ldd -r problem but then we're back to the command line length problem that this code was added to solve:

...
   CC  mpn_extras/arm64/sqr_hard_pic.asm
   CC  mpn_extras/arm64/sqrhigh_hard_pic.asm
Building libflint.21.0.0.dll
make: cc: Argument list too long

If the question is whether to use -r or not then I think that the simplest fix/approach is just to say that there could be a configure-time check for whether the linker accepts -r. If it does then use it and if not then don't because it doesn't work.

That won't help with flintlib/python-flint#360 though because we need a third option somehow. Passing -r to the linker doesn't work but having the long command line doesn't work either.