No permission asked, nothing happens

When connecting the switch to my phone nothing happens, I don't get a permission request or anything. Yes the switch is in RCM and the cable works.
I can't switch to have my phone controlling the USB device for some reason, that might be it.