Running make with `-j` causes issues in docker environments

[rzane]

Describe the problem as clearly as you can

After upgrading to RubyGems v4, the bundle install step in our CI runs started getting killed by the OS. Eventually, we found this change, which makes RubyGems run make with the -j flag.

In Docker environments, Etc.nprocessors often reports the wrong number of CPUs. Etc.nprocessors was reporting the number of available CPUs on the host machine, which was much larger than the number of CPUs that are actually available. As a result, the install process was getting killed.

The current workaround is to explicitly configure MAKEFLAGS like this:

export MAKEFLAGS=-j2

We found it very surprising and unintuitive that the number of make jobs is not synchronized with Bundler's jobs configuration.

I propose reverting this change, since it's going to cause a lot of headaches. Instead, I think this change should be implemented in Bundler, which already has a jobs setting.

Alternatively, documenting this workaround and calling it out explicitly in the changelog would have been tremendously helpful for us.

Did you try upgrading RubyGems?

I encountered this issue on the latest version of rubygems.

Post steps to reproduce the problem

Which command did you run?

I ran bundle install.

What were you expecting to happen?

I was expecting all parallelization during bundle install to be configurable with bundle config set jobs $NUMBER.

What actually happened?

Setting bundle config set jobs $NUMBER limited bundler's concurrency, but it did not limit make's concurrency.

Run gem env and paste the output below

[Edouard-chin]

Thank you for the bug report and sorry about this issue. Having the same number of jobs as BUNDLE_JOBS make sense to me. I'll open a fix

[rzane]

No need to apologize! All improvements come with some risk, and parallelizing make is a valuable improvement.

[hsbt]

In my macOS environment, Etc.nprocessors returns correct numbers.

❯ sysctl -n hw.ncpu
10
❯ sysctl -n hw.physicalcpu
10
❯ container run -it -a amd64 --rm ruby:3.4 bash
root@d1cb3173-43cf-4eb7-ac0e-69ffe8e2eecd:/# ruby -retc -e "puts Etc.nprocessors"
4

@rzane Can you share your environment or platform?

[mame]

Are you sure that Etc.nprocessor is returning an incorrect value in a Docker instance?

I have another hypothesis: Does bundle install invoke make commands in parallel? If bundler launches M instances of make -jN in parallel, it spawns at most N * M processes, which could lead to the catastrophe. If this hypothesis is correct, I think the proper fix would be for bundler to implement a jobserver for make.

https://www.gnu.org/software/make/manual/html_node/Job-Slots.html

[rzane]

We're running on CircleCI, where this is a known issue, but here are a few issues:

• https://circleci.canny.io/cloud-feature-requests/p/have-nproc-accurately-reporter-number-of-cpus-available-to-container
• https://support.circleci.com/hc/en-us/articles/10653544013467-Docker-Executor-returning-Total-Host-CPU-Cores
• https://travis-ci.community/t/nproc-reports-32-cores-on-arm64/5851
• Not correct os.cpus().length inside the docker container with cpus limited. nodejs/node#28762

The issue stems from using Docker with resource limits:

$ docker run -it --rm --cpus=1 ruby:3.4 bash
root@809feaec1e61:/# nproc
8
root@809feaec1e61:/# ruby -retc -e "puts Etc.nprocessors"
8

Lots of tools automatically parallelize based on the number of CPUs, so we're accustomed to explicitly setting concurrency limits in our CI jobs.

[Edouard-chin]

Thanks folks. So I think there are two distinct issues now.

1. Reuse the BUNDLE_JOBS value to let users control the number of jobs during compilation. This PR addresses that.
2. As @mame mentioned, bundle install invokes make in parallel. This is due to Bundler installing gems in parallel. I'll read about make jobserver and how to implement it as I have never heard of it.

@hsbt If you'd like to revert the MAKEFLAGS change while I fix those issues, feel free to do so. 2) may take me a bit of time to figure out.

[mame]

@rzane --cpus=1 limits CPU usage, not the number of CPUs. What you should use here is --cpuset-cpus. On my 12-core machine, it correctly limits as follows:

$ docker run -it --rm --cpuset-cpus 0,1 ruby:3.4 bash
root@2c5fe5adab7e:/# nproc
2
root@2c5fe5adab7e:/# ruby -retc -e "puts Etc.nprocessors"
2
root@2c5fe5adab7e:/#
exit
$ docker run -it --rm --cpuset-cpus 0,1,2,3 ruby:3.4 bash
root@07013be981c8:/# nproc
4
root@07013be981c8:/# ruby -retc -e "puts Etc.nprocessors"
4

[rzane]

It's quite possible that my understanding of how CircleCI works behind the scenes doesn't match reality. On CircleCI, I do not control how the docker image is launched. However, I am certain that docker containers on CircleCI will return the wrong number. In my case, nproc returns 32, but the CircleCI instance is a small, which has only one CPU.

Setting CircleCI aside, just because 8 CPUs are available, doesn't mean I want make to use all available resources. In a shared compute environment, you might want to avoid having gem installation affect other processes. In my opinion, there should be a documented way to limit concurrency.

[hsbt]

@rzane

Thanks. I and @mame discussed about this yesterday. We may need to provide new method that returns the virtual number of CPUs available on the container if we get that from syscall or something.

@Edouard-chin

I'm okay to the current implementation because workaround is available. But It would be better to add feature to specify job number like --jobs option.

The best way is to assign cores in make job server based on the number of BUNDLE_JOBS, the number of jobs to assign to gem install , and the number of CPUs you actually have available.