kernprof -m and -p cannot find modules inside a namespace package

[gwerbin-tive]

src/
  my_namespace/
    a/
      __init__.py
      core.py

PYTHONPATH="${PWD}/src" uv tool run --from 'line-profiler==5.0.0' kernprof -l -m my_namespace.a.core

I expected that this would work. But I get this error:

`Could not find module `my_namespace.a.core`

Adding src/my_namespace/__init__.py fixes the error and profiling appears to work normally.

[TTsangSC]

(For a moment I was confused since I distinctively remember there being a test for namespace packages with the function handling kernprof -p (the flag specifying the profiling targets). But I guess that was probably a typo and you're referring to kernprof -m (the flag for running modules), as indicated by the example.)

The issue seems to be with the static-analysis tools we're using. In line_profiler/autoprofile/util_static.py:

• While the function modpath_to_modname() comes with a check argument (which allows for disabling the check for __init__.py), the inverse function modname_to_modpath() used here doesn't.
• The latter function calls _syspath_modname_to_modpath(), which contains hard-coded checks for the existence of a __init__.py in the resolved directory.

The fix (?) should probably involve:

• Adding the corresponding check switch parameter to ~.autoprofile.util_static.modname_to_modpath(),
• Reconsidering whether it should default to true, seeing that namespace packages have been a thing since PEP 420 (i.e. Python 3.3), and
• Maybe just use the full import system (importlib.util.find_spec()) instead of static analysis for finding the source file associated with the module name.

Whatever fix/enhancement/behavior change we end up implementing with util_static should probably also be backported to ubelt and xdoctest. @Erotemic what would you say would be the most sensible approach here?

[Erotemic]

the most sensible approach here?

The most sensible approach? Don't use namespace packages because they're an awful idea. Static analysis can't reliably detect them.

Adding src/my_namespace/init.py fixes the error and profiling appears to work normally.

And now its clear to everyone that is a package! Code improved!

But yeah... I guess we could look into what importlib can help us with. There is a commented out _importlib_modname_to_modpath in ubelt's modname_to_modpath with:

def _importlib_modname_to_modpath(modname):  # nocover
    import importlib.util
    spec = importlib.util.find_spec(modname)
    modpath = spec.origin.replace('.pyc', '.py')
    return modpath

def modname_to_modpath(modname, hide_init=True, hide_main=False, sys_path=None):
    """
    Finds the path to a python module from its name.

    Determines the path to a python module without directly import it

    Converts the name of a module (__name__) to the path (__file__) where it is
    located without importing the module. Returns None if the module does not
    exist.

    Args:
        modname (str):
            The name of a module in ``sys_path``.

        hide_init (bool):
            if False, __init__.py will be returned for packages.
            Defaults to True.

        hide_main (bool):
            if False, and ``hide_init`` is True, __main__.py will be returned
            for packages, if it exists. Defaults to False.

        sys_path (None | List[str | PathLike]):
            The paths to search for the module.
            If unspecified, defaults to ``sys.path``.

    Returns:
        str | None:
            modpath - path to the module, or None if it doesn't exist

    Example:
        >>> from ubelt.util_import import modname_to_modpath
        >>> modname = 'xdoctest.__main__'
        >>> modpath = modname_to_modpath(modname, hide_main=False)
        >>> assert modpath.endswith('__main__.py')
        >>> modname = 'xdoctest'
        >>> modpath = modname_to_modpath(modname, hide_init=False)
        >>> assert modpath.endswith('__init__.py')
        >>> # xdoctest: +REQUIRES(CPython)
        >>> modpath = basename(modname_to_modpath('_ctypes'))
        >>> assert 'ctypes' in modpath
    """
    if hide_main or sys_path:
        modpath = _syspath_modname_to_modpath(modname, sys_path)
    else:
        # import xdev
        # with xdev.embed_on_exception_context:
        # try:
        #     modpath = _importlib_modname_to_modpath(modname)
        # except Exception:
        #     modpath = _syspath_modname_to_modpath(modname, sys_path)
        # modpath = _pkgutil_modname_to_modpath(modname, sys_path)
        modpath = _syspath_modname_to_modpath(modname, sys_path)

    if modpath is None:
        return None

    modpath = normalize_modpath(modpath, hide_init=hide_init,
                                hide_main=hide_main)
    return modpath

I know the pkgutil version is deprecated, but maybe slotting in the _importlib_modname_to_modpath works. I remember there being an issue with it, but I don't seem to have a comment about it. It could be that it doesn't work with pytest, which would be less of a problem here than in xdoctest.

---

Looking into it a little, I'm reminded what some of the issues with find_spec side effects are. If you try to find a submodule, you end up importing the parent namespace:

python -c "if 1:
    import sys
    import importlib
    importlib.util.find_spec('numpy.linalg')
    print(sys.modules['numpy'])
    print(sys.modules['numpy'].sum)
"

Actually has numpy imported.

There are a few ways forward:

1. Our docs state autoprofile doesn't work with namespace packages.
2. we try using the importlib.util.find_spec variant here, maybe backporting it. It would certainly help to keep all those copied definitions of this tool in sync (and honestly, I don't know why importlib doesn't have similar tools, the spec stuff seems over-complicated to me, but I guess that's what you need if you have some magic registry of namespace packages instead of just using the filesystem.)

[Erotemic]

I just don't see a way forward with importlib.util.find_spec. The fact that it will import the parent package seems like a huge issue. I'm open to ideas, I certainly have opinions about namespace packages, but they are supported, and if we can support them I would like to, I just don't see a way to do so without causing other issues. I'm open to ideas.

I was playing with using importlib in ubelt for this function (importlib is orders of magnitude faster than the syspath approach!), but it still breaks pytest: Erotemic/ubelt#167 I care much more about correctness here.

At the very least we can add documentation that states namespace packages do not resolve with -p.

[TTsangSC]

Well the thing is that...

• We are already using importlib with -p. Remember that discussion we had about whether to use static analysis during ENH: more intuitive profiling-target selection #337? The final resolution was that we go with dynamic imports since we're already doing eager pre-imports anyway (which just imports the targets and add them to the profiler and doesn't require rewrites, so the importing of parent packages is not an issue) – with the caveat that we also leave a dev switch to go full-static in case that goes awry. And as mentioned in the my reply to OP, we already have a test showing that dynamic resolution (the default) picks up namespace packages and static analysis doesn't.
• As it is now, the original rewrite-based autoprofile framework is pretty much only used for the profiled script/module proper, and everything else is eagerly pre-imported and fed to the profiler post-import.
• OP probably made a typo in the issue title and the problem was with the -m flag, which does use static analysis to retrieve the source file from the module name. In that case, namespace packages are indeed not adequately handled. But I do agree with you that trying to run namespace packages is problematic, since there could easily be several files corresponding to some.package, and thus the behavior of __main__ is ill-defined, save for the edge case of a single such file.
• I'd say however that using importlib for resolving -m isn't all that bad – probably. Even if anything (incl. parent and sibling modules) is loaded into sys.modules, ultimately the code executed is the statically-rewritten copy of the required source as long as that is singly well-defined. Which would be the case for ordinary modules and packages, as well as single-location namespace modules (retrieved via ModuleSpec.origin). With pathological namespace modules .origin would've been set to None), and we can always just raise an error when that happens.

Unless the objective is to absolutely avoid executing any code that isn't the file that python -m <module> resolves to, using static analysis over dynamic resolution for that singular file probably won't have many drawbacks.

But then again as you've noted switching over to dynamic resolution does generally leave more room for things to go sideways, and util_static should probably be left untouched. The breaking Pytest part though I'm not so sure is related. Wasn't it the case just that 3.8 choked because tuple[...] and list[...] were used in place of typing.Tuple[...] and typing.List[...]?

Either way yes, if we decide to WONTFIX we should probably put that on the docs.

[Erotemic]

The 3.8 error was independent, that had to do with tests for adding dataclass support to ubelt.hash_data.

You're right that this looks like an issue with -m and not -p. In that case, the feature needs to have parity with python -m, and we should use the dynamic importlib mechanism. So I consider that a bug.

I am worried that the tests won't run with pytest, the use-cases for the xdoctest / ubelt static analysis are quite different than line-profiler. For the former packages, most times modpath_to_modname is invoked, its not because they are about to import the module, they are instead generating code or enumerating doctests in the file. So maybe line-profiler won't run into those pytest issues.

[TTsangSC]

I am worried that the tests won't run with pytest

Just trying to figure out what was wrong with your ubelt pipelines:

• Note that the second-to-last pipeline (#1086) actually passed, and it was the unrelated xcookie stuff (#1088) that broke it. Apparently COVERAGE_CORE=ctrace causes problems with 3.14 subinterpretors and on PyPy. I downloaded your branch, ran pytest locally on 3.14rc1, and the test suite passed without the envvar, and failed on the one same test as your CI pipeline. In Update ci for 314 #369 we're only setting COVERAGE_CORE=ctrace for this project because we needed compatibility with Cython, but it seems that it causes problems with some other aspects of the language and implementations. Maybe COVERAGE_CORE=ctrace should not have been a general thing for xcookie...
• Other than that, I also tried to remove the xdoctest SKIP directives for ubelt/util_static.py::_importlib_modname_to_modpath(), and the test does now pass. Seems that it can be un-skipped without issues now, and maybe the pytest errors you saw were actually due to other confounding factors...

Or am I completely misunderstanding your situation, and your remark about how _importlib_modname_to_modpath() "doesn't play nice with pytest" was not about ubelt itself but rather test collection by xdoctest? E.g. if we're going through the import system without directly using static analysis, doctests in non-importable modules (e.g. due to missing dependencies, as we ran into in Theelx#8) cannot be skipped before the code is potentially executed (e.g. if the package __init__.py imports from a child) and the error is raised?

So I consider that a bug.

Got it, will start working on a PR for that. Indeed I don't see this specific usecase we have here being problematic, if any code the profiled module uses or any of its parent packages fails to run anyway it'd be their fault not line_profiler's.

[gwerbin-tive]

Thanks for the detailed and quick replies! The title was in fact a mistake, but I do get the same error with either option, whether individually or together. PYTHONPATH=$PWD/src kernprof -p my_namespace.a.core some-script-wrapper.py also produces the namespace package error. So maybe it has to do with the combination of a namespace package with a nonstandard PYTHONPATH?

[TTsangSC]

As mentioned, we already had a test for catching namespace packages with the -p flag, so I'd have thought that it'd work just fine. But then in your example your namespace package is kinda nested deeper relative to sys.path, which (though unlikely) might've caused the error you saw. I'll give it a test and see if I can reproduce the problematic behavior.

Oh and of course the issue is now closed because of the fix we pushed for -m. If errors persist, we can always continue on a new issue.