Skip to content

Add option to balance jobs by example instead of by file #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
navidemad wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add option to balance jobs by example instead of by file #1

navidemad wants to merge 5 commits into from

Conversation

@navidemad
Copy link
Owner

@navidemad navidemad commented Sep 19, 2025 •
edited
Loading

Forked from : briandunn#105

We are getting sometime that error:

 connection closed (DRb::DRbConnError)
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:648:in 'DRb::DRbMessage#recv_request'
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:960:in 'DRb::DRbTCPSocket#recv_request'
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:1688:in 'DRb::DRbServer::InvokeMethod#init_with_client'
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:1700:in 'DRb::DRbServer::InvokeMethod#setup_message'
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:1660:in 'DRb::DRbServer::InvokeMethod#perform'
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:1780:in 'block (2 levels) in DRb::DRbServer#main_loop'
	from <internal:kernel>:168:in 'Kernel#loop'
	from /home/runner/_work/app/vendor/bundle/ruby/3.4.0/gems/drb-2.2.3/lib/drb/drb.rb:1776:in 'block in DRb::DRbServer#main_loop'

grk and others added 4 commits January 29, 2025 15:09
@grk
Add option to distribute jobs by example instead of by file
9d5627a 
The purpose of this change is to allow more optimal distribution when
the suite contains outlier files.

With enough cores on a CI run we noticed that the lower bound of the
duration of the full run was the runtime of the slowest file.

In order to be able to split by example, ExampleJobBuilder needs to load
all spec files and extract examples from them. As a side effect, this
unlocks more filtering options (such as by tag) which previously would
silently not work as expected.
@grk
Fix sub-optimal distribution of untimed examples
e2bee92
@grk
Make sure to read all IPC before de-marshaling
4821967
@navidemad
Refactor to use drb
f053906
@navidemad navidemad mentioned this pull request Sep 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@navidemad @grk