If callers need to cross tenant boundary their issuer and audience are going to be different.
Assuming the application trusts the issuer we could allow this scenario, which would unblock a big gap in the current AAD model - enable service principals defined in tenant A to call services in tenant B.