It happens typically when developer creates a unit test class which becomes the first one in the directory, and module manifest is updated to include that path to the unit test.
Later the unit test is found to be unnecessary and thusly removed, causing an empty unit test directory. This may happen frequently during refactoring. However, if the path in the module manifest is not removed to reflect that, it will complain and fail when doing build.
It is reasonable for package manager to have some flexibility dealing with empty directories such as warning in dev mode instead of throwing errors.