It seems like parts of the game are keeping references to the device - when the game makes a new device (when the resolution or framerate changes), there are about 18 references left.
I did a dirty hack where I kept releasing until it deleted, and the lag that Feno reported goes away. However it does cause the game to crash a little bit, so this is obviously not ideal.