10.0 RabbitMQ Change Changelog Notice Issue

[flyersa]

I am not the popular one, but i can live with that, so here is another one ;)

So, just giving a link to

https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability

which does not apply to any osism related installation should work or what is the idea about that changelog notice?

can we fix this to something more related to osism users?

e.g basically that says "stop everything", run reset state, redeploy etc.?

providing a link to something that references to stuff like "kolla-ansible genconfig" is not really a helpfull releasenote. You did a major release here,10.0 is also 2025.1- Why is the changelog so short handed? I mean 10.0 is a major release, more pride please for the awesome job you guys do ?

Also please note you have alot of major osism users now. The release notes, docs should reflect how to properly use, update this in relation to osism instead of a kolla reference imho. Obviously we will figure this out ourself, but this should not be the goal.

Also missing in the changelog, what was fixed etc.? Do you guys need help from the osism community? I am sure there are plenty of people who will and want to help here (us included).

There are alot of bugfixes in this releases, not even mentioned, such as the ISO Problem which got fixed, the trunk port issue fixed and so on.

Imho this release note should be more proper, highlight fixed issues and also give upgrade advice to existing osism users instead of just doing a useless reference to some kolla doc which people cannot even use directly. Correct me if im wrong here.

I mean this upgrade is a major interuption change. Its not just like "yeah change the queues" in rabbitmq, this is a upgrade which will cause problems if not done properly. And there is no mention at it at all.

Imho we have a huge info gap on osism atm. As example the last security issue, who got the info? people had to ask? stil no info about patches, even CVE should be out now etc. I understand OSISM also wants to make revenue, but i am not sure this is the way to go.

Also adding again here the idea to have people pre-test the upgrades before a release, what happened with that? :(

[berendt]

The release notes are a work in progress and not yet finished.

[berendt]

Added a note to avoid further confusions.

[berendt]

Regarding "Imho we have a huge info gap on osism atm. As example the last security issue, who got the info? people had to ask? stil no info about patches, even CVE should be out now etc. I understand OSISM also wants to make revenue, but i am not sure this is the way to go."

We rebuild and publish our images every 24h hours. Since the patches were all merged upstream a few days ago, they are included in the latest images we provide. They are also available to everyone, there are no restrictions on availability here.

• https://review.opendev.org/966871 (2024.1/caracal(keystone)) (there is a backport for this one)
• https://review.opendev.org/966073 (2024.2/dalmatian(keystone))
• https://review.opendev.org/966071 (2025.1/epoxy(keystone))

[flyersa]

Regarding "Imho we have a huge info gap on osism atm. As example the last security issue, who got the info? people had to ask? stil no info about patches, even CVE should be out now etc. I understand OSISM also wants to make revenue, but i am not sure this is the way to go."

We rebuild and publish our images every 24h hours. Since the patches were all merged upstream a few days ago, they are included in the latest images we provide. They are also available to everyone, there are no restrictions on availability here.

• https://review.opendev.org/966871 (2024.1/caracal(keystone)) (there is a backport for this one)
• https://review.opendev.org/966073 (2024.2/dalmatian(keystone))
• https://review.opendev.org/966071 (2025.1/epoxy(keystone))

the images you update explicity need always to be manually tagged to the last release, the current images of a release are not updating. So people are forced to explcility set a tag to as example 2024.2. This is is another issue out of this scope- If i run a osism version i would expect to also update the images with the fixes without specifying overrides as example. But existing images do not get updates, just tagged ones.

example? existing osism install, pull new images? no fix! setting override to 2024.2 , fix included. That is a major thing. Yes you build images with fixes but they cannot be used without doing a override, they are even on a complete different path on the registry. If i deploy a specific version i want osism apply -a pull $whatever to pull a version with the fix included. But you only update the version taged ones. Once there is a release, those are not upgrading with bugfixes,

And for the CVE, only internal people got this. This should be available to all. No mentioning at all etc. except internal to "first class" customers. Maybe we use it wrong i dont know anymore.... with this i mean a "run this to fix it" note as example.

With overrides i mean to explicitly set those in images.yml to make use of them

[flyersa]

Just an idea, maybe we can do a monthly osism meeting round or something with users who want to take part? similar to what we had before with SCS? Does not need to be long. 30 minutes, BAM. I would love this as a direct channel. OUG? Osism User Group? :)

Also please excuse my hard direct comments, i value what you do and osism turned out to be great product by now. We did alot to get this adopted by alot of people too. I really love what you do and also which is not visible to anyone. So i think it would be good to just connect everyone again, or maybe im just missing out.

[fkr]

@flyersa I think, that is a very good idea. Initially it was proposed to continue with Team IaaS, Team Container and Team OPS/IAM calls like during the project phase. Team Container and Team OPS/IAM (here with very little attendance, tbh) are still holding meetings - so maybe through you raising it, this could be revived for IaaS as well.

[jklare]

Hi @flyersa :) I migrated this from an issue to the discussions section, since I feel there is a lot of interesting input here, but at the same time this is very challenging to consume as a single issue. In addition to that, the discussion around the points you mentioned already started, so let's continue it as a discussion and see if we can define actionable issues/work items from here.
I like the idea of a monthly SCS Team IaaS Meeting and think we should schedule one for December. How about Wednesday, 03.12. 2pm?

[jklare]

added a PR here SovereignCloudStack/calendar#404

[fkr]

very cool. Thanks!

An (obvious) comment: From the one-time event and the participation an interest can only be partially derived - since not everybody can schedule their day to make sure to attend (for example: I will not be able to attend or to re-prioritize in order to make room for this on Dec 3rd 14:00). However I think it is good to just set a date & time instead of doing big rounds of questionnaires to find a slot. People wanting to attend, that can't should make sure to voice themselves (and we should add this to the communication of this slot).

[jklare]

agreed :)

[jklare]

will people automatically get notified about the new meeting or should i post something to any of the existing mailing lists? @fkr

[fkr]

will people automatically get notified about the new meeting or should i post something to any of the existing mailing lists? @fkr

no, there is no automation there.

Imho an e-mail to scs-tech@ would be good. Furthermore it should be mentioned in the scs-tech matrix room (and likely the OSISM one).

I will ask my colleague Lisa to advertise the IaaS Call as part of the Webinar this thursday as well.

[flyersa]

what about the release notes now? Anyone working on those updates? Any caveeats or specials we need to consider in upgrading from 9.4.0 to 10.0 ? I hesitated atm to update at all any environments because the communication dropped. And tbh if you need a break or something i completly understand this. But also as i said if you need help let us help. I will update our testbed and report on the findings.

[jklare]

Hi @flyersa, we are currently working on this, but are a bit short on time at the moment. In addition to that, we will work on a better communication towards the community as discussed before. For the moment, I don't think there is any way you can assist us in preparing the release, but hopefully, we can include more people in the future. Building a more inclusive design and development process will take some time, but I hope we can get there in the coming months.

[flyersa]

No Problem. Thank you for all the effort and the great work!

[flyersa]

I see 10.0 was relabeled to 10.0-rc1 and you even provide a migration script and good documentation. Great job. Will test it this week and provide feedback

[flyersa]

I upgraded to 10.0.0-rc1 today , so far so good, however that rabbitmq stuff didnt went very well as nova didnt work anymore, need to redo this on another test lab later, functional tests so far look good.

10.0.0-rc1 notes:

Upgrade performed from 9.4.0 to 10.0.0-rc1

RabbitMQ upgrade:

failed directly with enable all stable feauture flags again, this is something we often see. It only helps to completly purge this and reinstall rabbitmq for some reason or just ignore it, maybe some regression on our testbed from a prior upgrade. Its also safe to ignore, the container just runs fine and is upgraded

TASK [rabbitmq : Enable all stable feature flags] ******************************
fatal: [ctrl02]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "rabbitmq", "rabbitmqctl", "enable_feature_flag", "all"], "delta": "0:00:00.744500", "end": "2025-12-20 12:08:45.152652", "msg": "non-zero return code", "rc": 69, "start": "2025-12-20 12:08:44.408152", "stderr": "Error:\nmissing_clustered_nodes", "stderr_lines": ["Error:", "missing_clustered_nodes"], "stdout": "Enabling all feature flags ...", "stdout_lines": ["Enabling all feature flags ..."]}

the documentation and the migration osism commands for rabbitmq worked flawless

Nice to have:

• masakari and hacluster are now in proper release path on the registry, but stil not deployable without override in images.yml because it cant find a path for some reason: unknown: artifact kolla/release/2025.1/hacluster-corosync:2025.1 not found, needs explicit override to:

hacluster_pacemaker_remote_image_full: "registry.osism.tech/kolla/release/2025.1/hacluster-pacemaker-remote:2.1.6.20251208"
hacluster_corosync_image_full: "registry.osism.tech/kolla/release/2025.1/hacluster-corosync:3.1.7.20251208"
hacluster_pacemaker_image_full: "registry.osism.tech/kolla/release/2025.1/hacluster-pacemaker:2.1.6.20251208"
hacluster_pcs_image_full: "registry.osism.tech/kolla/release/2025.1/hacluster-pcs:2.1.6.20251208"
masakari_monitors_image_full: "registry.osism.tech/kolla/release/2025.1/masakari-monitors:19.0.1.20251208"
masakari_engine_image_full: "registry.osism.tech/kolla/release/2025.1/masakari-engine:19.0.1.20251208"
masakari_api_image_full: "registry.osism.tech/kolla/release/2025.1/masakari-api:19.0.1.20251208"

then it works, but maybe ok if its not supported anyway.

Please also add python-masakariclient and the python-watcherclient in the openstackclient container

After Upgrading and following the rabbitmq notes from the release notes i was unable to launch instances anymore, they got properly scheduled from placement but were stuck forever in pending without any indication in nova on the assigned hypervisors what was causing it, no error, nor warning, nothing. So i removed om_rpc_host and om_notify_vhost from kolla/configuration.yml and rolled everything out again after purging rabbitmq completly and reinstalling rabbitmq first.

I suspect something is not properly pulling from the queue in nova if this is specified.

[flyersa]

you guys are back from vacation? :)

[flyersa]

Another one, upgrading to 10.0.0-rc2

upgrade of grafana seems to kill existing dashboards