[BUG] Breaking changes to tenants in 2.19.4

[AlexRuiz7]

What is the bug?

In previous versions, such as 2.19.3, Opensearch uses an empty string ("") as the global tenant, both with multi-tenancy enabled and disabled.

However, because of the changes introduced in OpenSearch 2.19.4 (#5519), the global tenant changes to "null" when multi-tenancy disabled. As a result, the reports (and every custom asset) falls under a different tenant compared to the previous version, so they don't show for a user using the global tenant.

How can one reproduce the bug?

In OpenSearch 2.19.3, disable multi-tenancy in Dashboards and create, for example, some reports definitions. Note the tenant is set to "".

Then, update to OpenSearch 2.19.4. The reports won't show in the UI. Also, in the image below, we can see how requesting the list of reports definitions for the admin user under the global tenant does return an empty response.

This gives the impression that the data was lost during the upgrade. However, the data is still present in the indices, just under a different tenant.

curl -k -u admin:admin -X GET "https://localhost:9200/_cat/indices/.opendistro-reports*?v"

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .opendistro-reports-definitions cQPXRDRXS-SHj_ESdWVDwA   1   0          1            0        5kb            5kb
green  open   .opendistro-reports-instances   eyemwFcfQ8Og0vI6XsAZUw   1   0          2            0      6.7kb          6.7kb

Create a new report definition. Note the tenant is now set to "null".

Do you have any additional context?

As a workaround, we created a script to migrate the resources, by updating every document whose tenant is "" to "null".

curl --key "$CERTS_DIR/admin-key.pem" --cert "$CERTS_DIR/admin.pem" --cacert "$CERTS_DIR/root-ca.pem" \
  -XPOST "https://127.0.0.1:9200/.*/_update_by_query?scroll_size=10000" \
  -H 'Content-Type: application/json' \
  -d'
{                  
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "tenant"
          }
        },
        {
          "term": {
            "tenant": ""
          }
        }
      ]
    }
  },
  "script": {
    "lang": "painless",
    "source": "ctx._source.tenant = \"null\""
  }
}'

This is clearly not ideal, but does the trick.

[cwperks]

@AlexRuiz7 Do you have plugins.alerting.filter_by_backend_roles set to true?

I suppose the proper fix here is not to serialize user_requested_tenant where multi-tenancy is disabled.