I've seen it occasionally that the X server would appear to have frozen, usually when switching back to it (I am using two Xorg instances, through startx).
This happened occasionally and I could not reproduce it.
I am using envoy-exec in a script started from .xprofile, which runs in the background.
And now I've noticed that this also happens during login already when the gpg-agent.conf contains a non-existent entry for pinentry-program: pinentry-program /usr/bin/pinentry-qt4 (which recently moved to /usr/bin/pinentry-qt on Arch Linux).
I am using -t gpg-agent with envoy, via systemctl enable envoy@gpg-agent.socket.