Hey,
I was playing around with pygmsh and gmsh in general, and as mentioned, once you have removed entities altogether using remove() in common Geometry, it doesn't necessarily remove them from the physical queue. Specifically, this is the part I am talking about.
The only way to add something in the (Physical) queue for example, is through this function. Even after meshing for example this object or dictionary is not removed as expected. If I have to clear all geo (using gmsh.clear() for example) and then make another geometry, I can't add it into the physical part as this will have already messed up the IDs.
Should be an easy fix to see if I have removed entities using remove(), then that should also remove entities from any other queue that it is a part of.