RuntimeError: dictionary changed size during iteration

[AndrewSazonov]

=================================== FAILURES ===================================
____________________ test_Lattice_from_pars[hexagonal-rad] _____________________
[gw0] darwin -- Python 3.13.1 /Library/Frameworks/Python.framework/Versions/3.13/bin/python

value = [3, 3, 4, np.float64(1.5707963267948966), np.float64(1.5707963267948966), np.float64(2.0943951023931953)]
ang_unit = 'rad'

    @pytest.mark.parametrize("ang_unit", ("deg", "rad"))
    @pytest.mark.parametrize("value", basic_pars)
    def test_Lattice_from_pars(value: list, ang_unit: str):
        ref = [v for v in value]
    
        if ang_unit == "rad":
            value = [
                value[0],
                value[1],
                value[2],
                np.deg2rad(value[3]),
                np.deg2rad(value[4]),
                np.deg2rad(value[5]),
            ]
    
>       l = Lattice(*value, ang_unit=ang_unit)

tests/unit_tests/components/test_Lattice.py:123: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easycrystallography/Components/Lattice.py:106: in __init__
    self.angle_beta = angle_beta
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/Objects/ObjectClasses.py:327: in __setattr__
    super(BaseObj, self).__setattr__(key, value)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/global_object/hugger/property.py:92: in __set__
    return super().__set__(instance, value)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/Objects/ObjectClasses.py:350: in setter
    obj._kwargs[key].value = value
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/global_object/undo_redo.py:473: in wrapper
    global_object.stack.push(PropertyStack(obj, func, old_value, new_value, **kwargs))
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/global_object/undo_redo.py:191: in push
    command.redo()
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/global_object/undo_redo.py:337: in redo
    self._set_func(self._parent, self._new_value)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/Objects/variable/parameter.py:191: in value
    value = self._constraint_runner(self.builtin_constraints, self._scalar.value)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/Objects/variable/parameter.py:394: in _constraint_runner
    constained_value = constraint(no_set=True)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/Constraints.py:115: in __call__
    dependent_obj = self.get_obj(self.dependent_obj_ids)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/Constraints.py:157: in get_obj
    return self._global_object.map.get_item_by_key(key)
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easyscience/global_object/map.py:109: in get_item_by_key
    if item_id in self._store.keys():
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <WeakValueDictionary at 0x105468550>

    def keys(self):
        if self._pending_removals:
            self._commit_removals()
        with _IterationGuard(self):
>           for k, wr in self.data.items():
E           RuntimeError: dictionary changed size during iteration

/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/weakref.py:222: RuntimeError
=============================== warnings summary ===============================
tests/unit_tests/components/test_Lattice.py: 24 warnings
  /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easycrystallography/Components/Lattice.py:133: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments. To learn more, see the migration guide https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword
    matrix = np.array(matrix, dtype=np.float64).reshape((3, 3))

tests/unit_tests/components/test_Lattice.py: 24 warnings
  /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/easycrystallography/Components/Lattice.py:133: UnitStrippedWarning: The unit of the quantity is stripped when downcasting to ndarray.
    matrix = np.array(matrix, dtype=np.float64).reshape((3, 3))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/unit_tests/components/test_Lattice.py::test_Lattice_from_pars[hexagonal-rad] - RuntimeError: dictionary changed size during iteration

[rozyczko]

This is becoming more serious and more reproducible.
Most current builds on EDL fail on python 3.12 on (almost) all platforms with this issue
E.g.
https://github.com/easyscience/EasyDiffractionLib/actions/runs/13546539737
https://github.com/easyscience/EasyDiffractionLib/actions/runs/13474780626

[AndrewSazonov]

Right, this is a very frustrating issue that has appeared randomly. Given our large matrix of platform, architecture, and Python version combinations, it occurs in every build.

Previously, I only encountered one or two failed combinations at a time, and simply re-running those allowed the build to pass. However, we need to fix this issue as soon as possible.

[damskii9992]

Copied over from the PR:
Fixed the RuntimeError: dictionary changed size during iteration issue in the vertices method by implementing thread-safety: iterations handling modifications during garbage collection.
This seems to have been a concurrency problem with weak references, where the dictionary is being modified during iteration.

Take the attached script (.txt -> .py cause GitHub)
run_test_suite.txt

Run it in say, EasyReflectometryLib or corelib

This now runs fine.

Root Cause

The Map class uses weakref.WeakValueDictionary() to store object references. When:

1. generate_unique_name() is called (e.g., when creating a new Parameter)
2. It calls self.map.vertices() which returns list(self._store.keys())
3. During the iteration over .keys(), the garbage collector may run
4. GC cleans up objects with no remaining strong references
5. This removes entries from the WeakValueDictionary
6. Python raises RuntimeError: dictionary changed size during iteration

This is particularly likely to happen when:

• Many temporary Parameter objects are created (e.g., in Lattice.volume property)
• The GC runs during the iteration
• Multiple objects are being created/destroyed in quick succession