PythonFunctionGenerator: make compatible with linecache by matthiasdiener · Pull Request #295 · inducer/pytools

matthiasdiener · 2025-03-31T22:24:11Z

Needs:

PythonFunctionGenerator: add decorator support #294.

BEFORE MERGE: SQUASH AS APPROPRIATE.

Somewhat clumsy, but can be very useful for debugging/profiling. E.g., for the following line_profiler test:

from pytools.py_codegen import PythonFunctionGenerator

cg = PythonFunctionGenerator("np_test", args=(),
                decorators=["from line_profiler import profile", "@profile"])
cg("import numpy as np")
cg("a = np.random.rand(1000000)")
cg("b = np.random.rand(1000000)")
cg("c = np.random.rand(1000000)")

cg.get_function()()

Before:

$ export LINE_PROFILE=1
$ python t.py
<generated code for 'np_test'>
Timer unit: 1e-09 s

  0.16 seconds - <generated code for 'np_test'>:2 - np_test
Wrote profile results to profile_output.txt
Wrote profile results to profile_output_2025-05-14T162934.txt
Wrote profile results to profile_output.lprof
To view details run:
python -m line_profiler -rtmz profile_output.lprof
$ cat profile_output.txt
Timer unit: 1e-09 s

Total time: 0.163372 s

Could not find file <generated code for 'np_test'>
Are you sure you are running this program from the same directory
that you ran the profiler from?
Continuing without the function's contents.

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2
     3
     4         1  106040000.0    1e+08     64.9
     5         1   42078000.0    4e+07     25.8
     6         1    4235000.0    4e+06      2.6
     7         1    8531000.0    9e+06      5.2
     8         1    2488000.0    2e+06      1.5

  0.16 seconds - <generated code for 'np_test'>:2 - np_test

After:

$ export LINE_PROFILE=1
$ python t.py
<ipython-input- generated: 'np_test'>
Timer unit: 1e-09 s

  0.14 seconds - <ipython-input- generated: 'np_test'>:2 - np_test
Wrote profile results to profile_output.txt
Wrote profile results to profile_output_2025-05-14T163030.txt
Wrote profile results to profile_output.lprof
To view details run:
python -m line_profiler -rtmz profile_output.lprof
$ cat profile_output.txt
Timer unit: 1e-09 s

Total time: 0.136593 s
File: <ipython-input- generated: 'np_test'>
Function: np_test at line 2

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           @profile
     3                                           def np_test():
     4         1   90751000.0    9e+07     66.4      import numpy as np
     5         1   30456000.0    3e+07     22.3      a = np.random.rand(1000000)
     6         1    4251000.0    4e+06      3.1      b = np.random.rand(1000000)
     7         1    8473000.0    8e+06      6.2      c = np.random.rand(2000000)
     8         1    2662000.0    3e+06      1.9      a+b

  0.14 seconds - <ipython-input- generated: 'np_test'>:2 - np_test

Note: Since the generated source code is not saved in a file on disk (only in the linecache), re-running the profile via python -m line_profiler -rtmz profile_output.lprof will not show the line contents. The profile_output.txt text files will work correctly.

matthiasdiener · 2025-03-31T22:37:48Z

pytools/py_codegen.py


 class PythonCodeGenerator(CodeGeneratorBase):
-    def get_module(self, name=None):
+    def get_module(self, name=None, write_file=True):


Could maybe set this to False by default.

I think we can do this without ever writing a file, by simply poking data into the linecache directly:

https://github.com/python/cpython/blob/3.13/Lib/linecache.py#L11-L13

This seems to be at least vaguely supported; cache is not underscored.

There should of course at least be a warning if the "file name" is not unique/has been seen before.

That won't work for the line_profiler unfortunately, it requires the code in a file, even if it is in the linecache: https://github.com/pyutils/line_profiler/blob/1630e7c9a295ace2feb1d2b188e68f4d2833fb20/line_profiler/line_profiler.py#L194-L210

51ada01 implements a workaround that uses linecache directly (like you suggested), but also maintains compatibility with line_profiler.

( I added a warning, it is somewhat noisy though)

inducer · 2025-04-02T22:02:01Z

pytools/py_codegen.py

-        return f"generated_code_for_{self.name}"
+        # Note that the '<ipython-input-' prefix is for compatibility with
+        # line_profiler: https://github.com/pyutils/line_profiler/blob/1630e7c9a295ace2feb1d2b188e68f4d2833fb20/line_profiler/line_profiler.py#L194-L210
+        return f"<ipython-input- generated code for '{self.name}'>"


Please only use the work-around if line_profiler is in sys.modules.

Done in 68ed96a

inducer · 2025-04-02T22:02:35Z

pytools/py_codegen.py

-        return f"generated_code_for_{self.name}"
+        # Note that the '<ipython-input-' prefix is for compatibility with
+        # line_profiler: https://github.com/pyutils/line_profiler/blob/1630e7c9a295ace2feb1d2b188e68f4d2833fb20/line_profiler/line_profiler.py#L194-L210
+        return f"<ipython-input- generated code for '{self.name}'>"


While we're at it, maybe shorten the normal prefix to just generated: .

Done in 68ed96a

inducer · 2025-04-02T22:04:04Z

pytools/py_codegen.py

+        if name in linecache.cache:
+            from warnings import warn
+            warn(f"Overwriting existing generated code in linecache: '{name}'.",
+                 stacklevel=2)


Use stacklevel=3 if this is called through get_function. (Maybe use an underscored parameter to tell the two apart?)

Done in 68ed96a

matthiasdiener · 2025-04-02T22:38:13Z

Would you prefer this separate from #294? (I need #294 to test this properly)

inducer · 2025-04-02T23:50:44Z

Just merged #294 to help you out. It'd be nice to add types for the decorators parameter. It could also default to () to avoid the None special case.

matthiasdiener · 2025-04-03T00:00:26Z

Just merged #294 to help you out. It'd be nice to add types for the decorators parameter. It could also default to () to avoid the None special case.

Thanks, done in 2f01274

matthiasdiener · 2025-05-14T21:33:33Z

This is ready for review @inducer

inducer

Thanks! A few more tweaks, then this should be good to go.

pytools/py_codegen.py

inducer · 2025-05-26T20:08:19Z

@matthiasdiener I've reworked this quite a bit. Please take a look and let me know what you think.

pytools/py_codegen.py

matthiasdiener · 2025-05-26T21:57:06Z

LGTM, thanks. I tested this with line_profiler, and it works as expected.

use linecache directly only adjust filename if line_profiler loaded improve args fix maintain compatibility with pudb broaden line_profiler usage check more type annotations pyright ignores rename generated functions to avoid linecache warnings remove special pudb handling make sure names are unique fix bpr expand to support PythonCodeGenerator, refactor fixes remove special line_profiler handling Relevant PR (pyutils/line_profiler#341) has been merged Rework for linecache compatibility, improve tests Co-authored-by: Andreas Kloeckner <inform@tiker.net>

add NanKeyWarning

matthiasdiener changed the title ~~Funcgen linecache~~ PythonFunctionGenerator: make compatible with linecache Mar 31, 2025

matthiasdiener commented Mar 31, 2025

View reviewed changes

matthiasdiener requested a review from inducer April 2, 2025 21:25

inducer reviewed Apr 2, 2025

View reviewed changes

matthiasdiener force-pushed the funcgen-linecache branch from 0ecb4b7 to 68ed96a Compare April 2, 2025 22:35

matthiasdiener requested a review from inducer April 2, 2025 22:36

matthiasdiener force-pushed the funcgen-linecache branch from 3068df1 to 4cb51b4 Compare April 3, 2025 15:25

matthiasdiener marked this pull request as ready for review May 14, 2025 21:33

inducer force-pushed the funcgen-linecache branch from e567f10 to 43127bf Compare May 15, 2025 16:27

inducer reviewed May 15, 2025

View reviewed changes

pytools/py_codegen.py Outdated Show resolved Hide resolved

pytools/py_codegen.py Outdated Show resolved Hide resolved

pytools/py_codegen.py Outdated Show resolved Hide resolved

matthiasdiener force-pushed the funcgen-linecache branch from 8874bc0 to 04837f7 Compare May 15, 2025 19:11

matthiasdiener requested a review from inducer May 15, 2025 19:23

matthiasdiener marked this pull request as draft May 15, 2025 19:41

matthiasdiener force-pushed the funcgen-linecache branch from 6430ede to a8b198a Compare May 15, 2025 20:19

matthiasdiener marked this pull request as ready for review May 16, 2025 01:46

matthiasdiener mentioned this pull request May 16, 2025

[RFC] Broaden support for profiling generated code pyutils/line_profiler#341

Merged

inducer force-pushed the funcgen-linecache branch from 23cd926 to 743be46 Compare May 21, 2025 17:16

matthiasdiener commented May 26, 2025

View reviewed changes

pytools/py_codegen.py Outdated Show resolved Hide resolved

matthiasdiener commented May 26, 2025

View reviewed changes

pytools/py_codegen.py Outdated Show resolved Hide resolved

inducer force-pushed the funcgen-linecache branch from de65183 to 1425f75 Compare May 26, 2025 20:13

matthiasdiener and others added 4 commits May 27, 2025 14:28

Fix apt in bpr CI

64b8482

Add NanKeyWarning, silence where appropriate

2e86ea0

add NanKeyWarning

bpr: create a more lenient exec env for tests

0126675

Fire mypy

9722f81

inducer force-pushed the funcgen-linecache branch from 1425f75 to 9722f81 Compare May 27, 2025 19:29

inducer enabled auto-merge (rebase) May 27, 2025 19:30

inducer merged commit 209e3ac into inducer:main May 27, 2025
17 checks passed

matthiasdiener deleted the funcgen-linecache branch May 27, 2025 19:42

Conversation

matthiasdiener commented Mar 31, 2025 • edited by inducer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthiasdiener commented Apr 2, 2025

Uh oh!

inducer commented Apr 2, 2025

Uh oh!

matthiasdiener commented Apr 3, 2025

Uh oh!

matthiasdiener commented May 14, 2025

Uh oh!

inducer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

inducer commented May 26, 2025

Uh oh!

Uh oh!

Uh oh!

matthiasdiener commented May 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matthiasdiener commented Mar 31, 2025 •

edited by inducer

Loading