SAP HANA Scale-Out cluster unexpectedly demotes primary following probe operations

[Bakerj0]

Opened a SAPHanaSR-ScaleOut Issue, but not sure if this would be the right location, so opening this new one:

• SAP HANA Scale-Out cluster unexpectedly demotes primary following probe operations SAPHanaSR-ScaleOut#143

During probes when saphana_monitor_clone() operation is ran, we run crm_resource command to check which node should be master, all other nodes are marked as DEMOTED:

saphana-controller-lib Line 1782

The crm_resource -W -r "$OCF_RESOURCE_INSTANCE" command however, will not report any node currently undergoing a probe operation, and this command will skip outputting any node doing a probe ( by design ):

# No output from rhel8-node2, which is current mastert:
$ cat SAPHana_RH1_HDB01.monitor.2025-08-12.11:04:15
-----------------------------------------8<----------------------------------------- 
++ 10:54:04: saphana_monitor_clone:3109: crm_resource -W -r SAPHana_RH1_HDB01
+ 10:54:04: saphana_monitor_clone:3109: super_ocf_log info 'DBG PROBE: 
resource SAPHana_RH1_HDB01 is running on: rhel8-node4
resource SAPHana_RH1_HDB01 is running on: rhel8-node5
resource SAPHana_RH1_HDB01 is running on: rhel8-node3
resource SAPHana_RH1_HDB01 is running on: rhel8-node2'

The way this is written guarantees that all probe operations will see all nodes as "DEMOTED". Probes were resources are active as master are rare, since this special monitor type really only runs on cluster start and were everything is already demoted. In rare cases were probes are run on a node with active master though, this can result in a demote against current master in some cases.

This issue is reproducible by creating any probe against the promoted instance of the SAPHanaController resource, but this can be tricky as probes can be difficult to generate.

The "pcs resource refresh" command while cluster is in a managed state should work or with following steps, can be run to generate a probe against just the primary:

1. Create any monitor failure against the "SAPHanaController" primary resource, while replication is down, so the primary is recovered to the same node.

2. Following recovery, run a "pcs resource cleanup" to clear errors. This will also run probes against resources with previous failures to check current status.4.

I have tested with using crm_attribute instead, and checking against the promotion score, and have found that crm_attribute is not susceptible to the same issue. Below change bypasses the issue:

# Updating RA as below:
    if ocf_is_probe; then
        super_ocf_log info "DEC: PROBE ONLY"
        # TODO PRIO1: NG - could check the Master status in a probe - after a refresh this status should be lost?
        # TODO PRIO1: NG - score, if CLONE_ATTRIBUTE is known 
        # TODO PRIO1: NG - is 'Master' still correct for promoted clones?
        #
        # check during probe, if this instance is *NOT* running as master
        # setting clone_state to "DEMOTED" is needed to avoid misleading "PROMOTED"/"PROMOTED"
        #
        crm_res=$(crm_resource -W -r "$OCF_RESOURCE_INSTANCE")
        [[ "$crm_res" =~ "is running on: "(.+)" "(Promoted|Master)"" ]] && master_node_name="${BASH_REMATCH[1]}" || master_node_name=""   <----
        if [ "$master_node_name" != "$NODENAME" ]; then   <----
. -----------------------------------------8<----------------------------------------- 
         node_score=$( crm_attribute --promotion="$OCF_RESOURCE_INSTANCE" -G | awk -F 'value=' '{ print $2 }' ) #<---- To this   
        if [ "$node_score" -ne 150 ]; then                          # <----

[fmherschel]

@Bakerj0 Could you explain the real problem behind? On SUSE SLES the RAs are working for more then 10 years (classic) and 2 years (angi). The DEMOTE-on-all-instances attribute could be a cosmetic but in some maintenance cases this is exactly what the real (cluster perspective) situation is. The cluster will probably run a promote on the primary master nameserver again but that is an "empty" action. SAPHanaController (angi) will detect the already as primary running master namerserver and does nothing beside setting the (cosmetic) attribute to "PROMOTED".
In our automated Lab tests we run a huge number of tests and we are tracking, if the cluster behaves correctly and we do not identify a cluster problem (misbehavior) in case of a probe run (e.g. refreshing the resource).

[fmherschel]

@baker0 In sum you say using crm_attribute does return an other value as crm_resource does? The 150 (score) does also not really say that the resource is REALLY promoted. And using a static 150 is also not wise as the scoring tables are developing. So 150 is NOT guaranteed.

[Bakerj0]

@baker0 In sum you say using crm_attribute does return an other value as crm_resource does? The 150 (score) does also not really say that the resource is REALLY promoted. And using a static 150 is also not wise as the scoring tables are developing. So 150 is NOT guaranteed.

@fmherschel thanks for your help and feedback! This is correct. To be specific crm_resource, won't report any nodes resources currently undergoing a probe. Since this command is specifically run exclusively during a probe, it guarantees that we won't the expected Master node during this special monitor operation. Most probes should be ran only on cluster startup, when there are no promoted instances so this won't be an issue most the time. But if somehow a probe is run against a master node ( example case in description ), this can lead us setting the promote attribute as "DEMOTED" incorrectly.

@Bakerj0 Could you explain the real problem behind? On SUSE SLES the RAs are working for more then 10 years (classic) and 2 years (angi). The DEMOTE-on-all-instances attribute could be a cosmetic but in some maintenance cases this is exactly what the real (cluster perspective) situation is. The cluster will probably run a promote on the primary master nameserver again but that is an "empty" action. SAPHanaController (angi) will detect the already as primary running master namerserver and does nothing beside setting the (cosmetic) attribute to "PROMOTED".
In our automated Lab tests we run a huge number of tests and we are tracking, if the cluster behaves correctly and we do not identify a cluster problem (misbehavior) in case of a probe run (e.g. refreshing the resource).

A customer had observed the issue discussed in the below article ( link and pdf attached ), where a promoted node running this probe will mark as "DEMOTED", and then subsequent standard monitors in the cluster for the resource would return a standard success code of OCF_SUCCESS ( 0 ), instead of the expected promoted success code of OCF_RUNNING_MASTER ( 8 ). For a promoted instance, the OCF_RUNNING_MASTER is expected for a promoted resource and a return of unpromoted success code '0' is treated as a monitor failure. So due to the set of DEMOTED in hana_<sid>_clone_state and subsequent "unpromoted success code" this did result in the demotion of a valid promoted instance and additional recovery in customer scenario:

• SAP HANA Scale-Out cluster unexpectedly demotes primary following probe operations.pdf

• SAP HANA Scale-Out cluster unexpectedly demotes primary following probe operations

The issue would be extremely rare, since probes don't normally run automatically on already active instances. The customer in this scenario was able to reproduce the probe on the primary, simply by doing a reboot ( echo b > /proc/sysrq-trigger ) against one of the other ( secondary ) nodes in the cluster. I am still not wholly sure why that action on a secondary triggered a probe on the primary though, and wasn't able to recreate that particular behavior myself and it seems uncommon ( although not a BUG in itself ). To create a similar probe during testing in primary, I had to take the steps attached in description ( create a previous monitor failure / failcnt on primary, and then use pcs resource cleanup while a failcnt was present on primary ).

To help track crm_resource output during these probes, we added below line to just report the full output of the command, as output didn't show up in regular traces. This helped to reflect the missing nodes, that didn't show due to probes, but otherwise should have been available:

function saphana_monitor_clone() {
-----------------------------------------8<----------------------------------------- 
    if ocf_is_probe; then
        super_ocf_log debug "DBG: PROBE ONLY"
        super_ocf_log info "DBG PROBE: $( crm_resource -W -r $OCF_RESOURCE_INSTANCE )"  #<--- Add this line

I did spend some time reviewing crm_resource code on my own, and with Pacemaker maintainers, and this appeared to be the design for this command ( to hide output from nodes undergoing probes ). This is the feedback they had provided here:

The problem here, based on my understanding of the issue your customer is seeing: doing a resource cleanup clears that resource's history from the CIB. Then a probe happens in order to check the resource's current status and repopulate the resource history.
For this resource agent, that very probe operation is looking for information that was deleted by the same event that triggered the probe (i.e., the resource cleanup).

It looks to me as if this is expected behavior for crm_resource -W. At least the way it's currently implemented. It's not ideal, because it's not actually telling us where the resource is running -- it's telling us where the current CIB thinks the resource is running.

Natural question: "Why don't we have crm_resource -W find out where the resource is actually running?" Answer: We would do that with a probe operation. And then we've come full circle back to the original problem.

It does seem like it would be a low frequency issue where we run probes on these promoted instance, and even more rare where it triggers an issue, but it does seem to occur in some cases. Due to this I thought crm_resource may be better replaced by another option such as crm_attribute which doesn't show same issue, or another tool which may be used to report the master.

Thanks again for your help here!

[fmherschel]

Could you share the effective configuration of the SAPHanaController and SAPHanaTopology resource (both the resources and their clones / master resources)? Do you have limited the number of clone instances per node?

[fmherschel]

Seams to be fixed already in SAPHanaSR-angi. The customer should migrate to SAPHanaSR-angi.
Tested on SUSE SLES15 with SAPHanaSR-angi version 1.2.12.