Skip to content

Conversation

@rus-max
Copy link
Collaborator

@rus-max rus-max commented Dec 26, 2025

@rus-max
Copy link
Collaborator Author

rus-max commented Dec 26, 2025

Hi!
Could you please help me with the failed pipeline?
@FooBarWidget

@FooBarWidget
Copy link
Member

FooBarWidget commented Dec 29, 2025

@rus-max I think Github Actions is failing due to running out of disk space. I see this in attempt 4, there was an error saying "no space left on device". Looks like Github Actions runner itself fails due to running out of disk space which is why you don't even see any logs after the event.

This is a known limitation of the current approach, and one of my fears with #178, which really increases disk space stress. We haven't invested in a structural solution for a long time but it seems we've finally hit the limit.

A real, robust, long-term solution would be to switch to an APT management system that doesn't rely on the local disk and can operate directly on cloud storage. I don't think such a solution exists, which means we'll have to write our own (not for the faint of heart).

Short-term workarounds might be possible, but we don't know until we try. They don't really solve the underlying limitation, but kick the can down the road, which means we'll run into the problem some time in the future again.

  • Clean Github Actions runner disk space more aggressively. I don't know how though: I think we're already cleaning everything we can.
  • Remove unnecessary packages from the APT repository. Some thoughts here:
    • We can remove all but the latest package revisions.
      • For example if there's fullstaq-ruby-2.5.7-jemalloc_0 and fullstaq-ruby-2.5.7-jemalloc_1, we can get rid of the _0 revision since it has absolutely no value anymore. Nobody's going to use an older package revision.
      • This requires a surgical modification on the APT repo directly. This means downloading the APT repo version folder from Google Cloud Storage, using aptly to remove the relevant packages from the database and publishing the changes to a local folder, then uploading the result back to Google Cloud Storage as a new APT repo version folder.
    • We can remove the malloctrim variant for Ruby >= 3.3 since Ruby natively has it.
      • This means people who installed the malloctrim variant for Ruby >= 3.3 won't get auto-upgrades anymore, and their existing 'apt-get install' automations will fail, but I think that's an acceptable tradeoff if that can give us more breathing space. We'll just have to communicate this to users well.
      • This will involve surgically removing relevant packages from Google Cloud Storage, like what I described above.
      • It also means modifying the build system to no longer build malloctrim variant for Ruby >= 3.3.
  • Some other workaround that gives us more disk space? Maybe there's some block storage we can mount? Or maybe we can mount Azure Blob Storage as an ephemeral local filesystem through FUSE, and hope that Aptly works properly with that? For example something like JuiceFS. I expect performance to be pretty bad though.

I would really like to work on the long-term solution, but unfortunately my health doesn't allow me to at the moment.

@FooBarWidget
Copy link
Member

FooBarWidget commented Dec 29, 2025

@rus-max So let me know how you would like to proceed and whether you need anything more from me. Access can be arranged. With my current health, I am in no condition to do any of that work myself. Or perhaps you can rally the community for help.

@rus-max
Copy link
Collaborator Author

rus-max commented Dec 30, 2025

I'm ready to take responsibility for starting with the simple approach — freeing up space for new builds.

Get well soon and come back stronger!

@FooBarWidget

@FooBarWidget
Copy link
Member

Thanks! I've sent you an email with additional setup instructions.

@FooBarWidget
Copy link
Member

Oh looks like your Git email address is not valid. What's your email address? If you don't want to post it here in public, you can email me instead. See https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/members.md

@rus-max
Copy link
Collaborator Author

rus-max commented Dec 30, 2025

Oh looks like your Git email address is not valid. What's your email address? If you don't want to post it here in public, you can email me instead. See https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/members.md

russs.max@gmail.com
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants