Support for Python 3.12

[tonybaloney]

This adds support for Python 3.12 (so far, the release is months away).

PRECALL and LOAD_METHOD have been removed. So the if-macro that says version >= 3.11 would be invalid for all future releases.

JUMP_IF_TRUE_OR_POP and JUMP_IF_FALSE_OR_POP have been removed.

[tonybaloney]

Almost working, it seems to be mixing up the CALL opcode so it's trying to call len on self (the module) even though it's a builtin

python ../atheris/example_fuzzers/custom_mutator_example.py
python(31049,0x7ff847b44340) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Using preloaded libfuzzer
INFO: found LLVMFuzzerCustomMutator (0x10bcbbad0). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1565970448
INFO: Loaded 1 modules   (16677 inline 8-bit counters): 16677 [0x10c204b88, 0x10c208cad),
INFO: Loaded 1 PC tables (16677 PCs): 16677 [0x10c208cb0,0x10c249f00),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 45 ft: 45 corp: 1/1b exec/s: 0 rss: 91Mb

 === Uncaught Python exception: ===
AttributeError: module 'atheris' has no attribute 'len'
Traceback (most recent call last):
  File "/Users/anthonyshaw/projects/cpython/../atheris/example_fuzzers/custom_mutator_example.py", line -1, in TestOneInput
AttributeError: module 'atheris' has no attribute 'len'

==31049== ERROR: libFuzzer: fuzz target exited
    #0 0x108cd24a5 in __sanitizer_print_stack_trace+0x35 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x544a5) (BuildId: 756bb7515781379f84412f22c4274ffd2400000010000000000a0a0000030d00)
    #1 0x10c1d1db8 in fuzzer::PrintStackTrace() FuzzerUtil.cpp:210
    #2 0x10c1b3d0c in fuzzer::Fuzzer::ExitCallback() FuzzerLoop.cpp:250
    #3 0x7ff804342ba7 in __cxa_finalize_ranges+0x19f (libsystem_c.dylib:x86_64+0x2aba7) (BuildId: 0773ddbc707e3b56ad3e97aaa9b2c3ed32000000200000000100000000030d00)
    #4 0x7ff8043429ba in exit+0x22 (libsystem_c.dylib:x86_64+0x2a9ba) (BuildId: 0773ddbc707e3b56ad3e97aaa9b2c3ed32000000200000000100000000030d00)
    #5 0x10792e93f in Py_Exit pylifecycle.c:2988
    #6 0x107948778 in _PyErr_PrintEx pythonrun.c
    #7 0x107945995 in _PyRun_SimpleFileObject pythonrun.c:439
    #8 0x107944771 in _PyRun_AnyFileObject pythonrun.c:78
    #9 0x1079c0096 in Py_RunMain main.c:689
    #10 0x1079c1344 in pymain_main main.c:719
    #11 0x1079c1657 in Py_BytesMain main.c:743
    #12 0x7ff80411741e in start+0x76e (dyld:x86_64+0xfffffffffff6e41e) (BuildId: f22a114397323e23a8b7cbade6bb830132000000200000000100000000030d00)

SUMMARY: libFuzzer: fuzz target exited
MS: 1 Custom-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0x78,0x9c,0xf3,0xc8,0x4,0x0,0x0,0xfb,0x0,0xb2,
x\234\363\310\004\000\000\373\000\262
artifact_prefix='./'; Test unit written to ./crash-34d8a0eeba0ec73df6e771631fc49f68dedfc122
Base64: eJzzyAQAAPsAsg==

[AidenRHall]

Thanks for writing this up Tony! Our project is definitely sensitive to these kinds of changes and they can be somewhat tricky to debug. Please let us know if we can help in any way :)

[n-bes]

Python 3.13 is coming. Any updates here?

[ranvit]

@tonybaloney @AidenRHall

I think we need to match opcode_caches against the definition of _PyOpcode_Caches in cpython 3.12, seen here

so its not sufficient to delete PRECALL, we need to update a bunch of opcodes' sizes.

And its changing even further in cpython 3.13

I'm not familiar with cpython, so

• idk why the opcode_cache is being redefined in this package
• idk if theres other cpython internals being redefined, that also need to be kept in sync across python versions
• idk if there's a more scalable way to keep this package sync'd -- perhaps a build step that automates the retrieval of opcode cache sizes and any other cpython internals?

Or, you could stop supporting backwards compatibility and update src/native/codetable_gen.cc per cpython version? Or throw a bunch of if/else at it.

Hope this helps someone get started on the enhancement! 🫡

[mingxiaoshan123]

hello, how to deal with code instrument error for Python 3.12.8:

Traceback (most recent call last):
  File "/home/xiaoju/voyager/simflow/fuzzing_url.py", line 15, in <module>
    @atheris.instrument_func 
     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xiaoju/python-3.12/lib/python3.12/site-packages/atheris/instrument_bytecode.py", line 1176, in instrument_func
    func.__code__ = patch_code(func.__code__, True, True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xiaoju/python-3.12/lib/python3.12/site-packages/atheris/instrument_bytecode.py", line 1157, in patch_code
    inst.trace_str_flow()
  File "/home/xiaoju/python-3.12/lib/python3.12/site-packages/atheris/instrument_bytecode.py", line 1043, in trace_str_flow
    elif self._is_str_hookable(
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xiaoju/python-3.12/lib/python3.12/site-packages/atheris/instrument_bytecode.py", line 930, in _is_str_hookable
    and self._names[instr.arg] in ("startswith", "endswith")
        ~~~~~~~~~~~^^^^^^^^^^^
IndexError: list index out of range

Log

[INFO][2024-12-25T15:19:10.528+0800][instrument_bytecode.py:927] _undef||msg=_is_str_hookable: instr.mnemonic=LOAD_ATTR, instr.arg=8, self._names=['decode', 'replace', 'client', 'get', 'status_code', 'print', 'RuntimeError', '_trace_branch']||appname=unkown||process=76382||thread=140015204730688||traceid=26585506896||client_ip=0.0.0.0||worker_id=-1

Thanks!

[mahdoosh1]

i modified src/native/codetable_gen.cc and it is building. i will let you know if it fails.

i am using termux to build it and it is amazing that it has not failed

[mahdoosh1]

it successfully installed.

changes i did:

• comment out `...[PRECALL] = 1;`
  

python version: 3.12.9
uname -a: Linux localhost 6.6.30-android15-8-g957ee129519c-4k #1 SMP PREEMPT Thu Feb 6 10:25:11 UTC 2025 aarch64 Android

python -c "import atheris": gives python version error

i will fix that, the fact that it works is enough for me

[AidenRHall]

Thx @mahdoosh1! It's great to see such engagement from end users. Maybe a little more transparency about where I'm at with debugging would help.

Unfortunately, what is described above is definitely not sufficient to get the fuzzer to actually run: there is also the problem of jump instruction offsets being calculated differently based on the number of CACHE instructions. From https://docs.python.org/3.12/library/dis.html "Changed in version 3.12: The argument of a jump is the offset of the target instruction relative to the instruction that appears immediately after the jump instruction’s CACHE entries."

I have the following code in my draft change to handle this, all within instrument_bytecode.py:

def get_cache_offset(i: int, instructions: List[dis.Instruction]) -> int:
cache_offset = 0
while i + 1 < len(instructions):
next_instruction = instructions[i + 1]
if next_instruction is None or next_instruction.opname != "CACHE":
break
cache_offset += 2
i += 1
return cache_offset

Then I've created a new attribute in the Instruction class called cache_offset, which is used to calculate self.reference in the ctor:

  self.reference: Optional[int] = (
      self.offset
      + self.get_size()
      + self.cache_offset
      + jump_arg_bytes(self.arg) * rel_reference_scale(self.mnemonic)
  )

and in the check_state method on the same class:

    assert (
        self.offset
        + self.get_size()
        + self.cache_offset
        + jump_arg_bytes(self.arg) * rel_reference_scale(self.mnemonic)
        == self.reference
    )

I modified str_fuzzing_example.py to include a for loop (which is what currently causes it to fail) to reproduce the breakages we're seeing in our other internal fuzzers. Even with this change the fuzzer still crashes with this error:

=== Uncaught Python exception: ===
TypeError: 'str' object is not callable
Traceback (most recent call last):
File: "...atheris/example_fuzzers/str_fuzzing_example.py", line None, in TestOneInput

The lack of debug info is...concerning. I am not sure why this is breaking and don't really have the tools to debug, because what I really need is to step through bytecode instruction by instruction, but there is no tooling support to do that internally (I have already reached out to the Python team internally to confirm this). So this is why it's stuck. I cannot emphasize enough how important it is that Atheris can run on Google's internal infrastructure, if we cannot do that then my fear is the project may ultimately be abandoned due to lack of business value to the company. So I can't just ignore the internal side of things.

One thing I haven't investigated enough is how this interacts with Google's internal Python infra and build system. Maybe that is the next step.

[mahdoosh1]

@AidenRHall that is unfortunate, it might be possible to rewrite some functions and classes that break in python >= 3.11; i am afraid i can't help, this is a big project (owned by big company google), and i have no experience on such projects, i will help you if i manage to make it work.

[AidenRHall]

The breakage I'm seeing is too general (it breaks if there's a for loop) to paper over with specific refactors - fuzzing instrumentation has to work in the general case. Of course I don't expect you to debug this (or Google's internal python infra), although I appreciate your efforts.

Big companies own lots of big projects, and unfortunately this one is small compared to the others - it is unrealistic to expect any additional resources get allocated to Atheris in the forseeable future, other than through me evangelizing Atheris internally to find teams who might want to use it for their projects.

[mahdoosh1]

i get

  File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/atheris/instrument_bytecode.py", line 1045, in trace_str_flow
    condition = self._is_str_hookable(
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/atheris/instrument_bytecode.py", line 930, in _is_str_hookable
    and self._names[instr.arg] in ("startswith", "endswith")
        ~~~~~~~~~~~^^^^^^^^^^^
IndexError: list index out of range

EDIT: i did not fix this. this is coming from an experimental function.

[mahdoosh1]

debug info:
lines where it happened:

60     >>   76 NOP
61          78 LOAD_FAST                1 (decompressed)
            80 LOAD_ATTR                9 (NULL|self + decode)  //error happened here, dis.dis says (NULL|self + decode)
            100 CALL                     0
            108 LOAD_CONST               3 ('FU')
            110 COMPARE_OP              40 (==)
            114 POP_JUMP_IF_FALSE       11 (to 138)
62          116 LOAD_GLOBAL             11 (NULL + RuntimeError)

[mahdoosh1]

it might be because of LOAD_FAST:

LOAD_FAST(var_num):
    Pushes a reference to the local co_varnames[var_num] onto the stack.
    Changed in version 3.12: This opcode is now only used in situations where the local variable is guaranteed to be initialized. It cannot raise UnboundLocalError.

[AidenRHall]

Well done @mahdoosh1! Love to see the engagement here. I suspect this is due to the change in how jump offsets are computed: https://docs.python.org/3/whatsnew/3.12.html#cpython-bytecode-changes, specifically this part:

Remove the LOAD_METHOD instruction. It has been merged into LOAD_ATTR. LOAD_ATTR will now behave like the old LOAD_METHOD instruction if the low bit of its oparg is set. (Contributed by Ken Jin in gh-93429.)

If you want to account for this I added these lines to version_dependant.py:

if PYTHON_VERSION <= (3, 11):
  def get_name(names, name):
    try:
      return names.index(name)
    except ValueError:
      names.append(name)
      return (len(names) - 1)

  def adjust_arg(arg: int):
    return arg

else:
  def get_name(names, name):
    try:
      return names.index(name) << 1
    except ValueError:
      names.append(name)
      return (len(names) - 1)  << 1

  def adjust_arg(arg: int):
    return arg >> 1

and then replace the calls inside instrument_bytecode.py with these functions. I hope that unblocks your debugging - it's actually really useful to have someone working on this on the OSS side, please let me know what your next error is if you continue pursuing this work. I am getting this one now, but I'm stuck on getting better debug info:

 === Uncaught Python exception: ===                                                                       
TypeError: 'str' object is not callable                                                                   
Traceback (most recent call last):                                                                        
  File ".../atheris/example_fuzzers/str_fuzzing_example.py", line None, in TestOneInput
TypeError: 'str' object is not callable

Part of the problem is the debug metadata isn't getting modified on my end, so I'm pretty sure this error message is misleading. I am gonna try messing with the modified version of str_fuzzing_example.py that I have locally to see if I can uncover more information about when it does or does not crash:

@atheris.instrument_func  # Instrument the TestOneInput function itself
def TestOneInput(data):
  """The entry point for our fuzzer.

  This is a callback that will be repeatedly invoked with different arguments
  after Fuzz() is called.
  We translate the arbitrary byte string into a format our function being fuzzed
  can understand, then call it.

  Args:
    data: Bytestring coming from the fuzzing engine.
  """
  fdp = atheris.FuzzedDataProvider(data)
  data = fdp.ConsumeString(sys.maxsize)
  allstrs = ""
  strl = ["foo", "bar", "baz", "biz"]
  for x in strl:
    allstrs += x
  # commented out code which I keep adding and removing in pieces
  """
  strs = iter(strl)
  for s in strs:
    allstrs += s
    allstrs += s
    if "c" in s:
      s += "d"
    if "a" in s:
      s += "b"
    allstrs += s
    s = str(reversed(s))
    if "c" in s:
      s += "d"
    if "a" in s:
      s += "b"
    if "c" in s:
      s += "d"
  """

  # This will be instrumented since the str startswith method is called
  # Note that this also works for the str endswith method as well
  if data.startswith("foobarbazbiz", 5, 20):
    raise RuntimeError("Solved str startswith method")


atheris.Setup(sys.argv, TestOneInput)
import dis; dis.dis(TestOneInput)
atheris.Fuzz()

Hope this helps!

[mahdoosh1]

Okay, now my crash is fixed, thank you for the fix. i will work on yours as well soon

[AidenRHall]

Ah the problem was that a CACHE instruction was being inserted after FOR_ITER which was overwriting the call to PUSH_NULL, so the state of the stack didn't conform to the calling convention. From the docs:

CALL(argc)
Calls a callable object with the number of arguments specified by argc, including the named arguments specified by the preceding [KW_NAMES](https://docs.python.org/3.12/library/dis.html#opcode-KW_NAMES), if any. On the stack are (in ascending order), either:

- NULL
- The callable
- The positional arguments
- The named arguments

or:

- The callable
- self
- The remaining positional arguments
- The named arguments

The bytecode in question looks like this:

        >>  154 FOR_ITER                25 (to 208)                                                                                                                                                                 
            158 LOAD_CONST               7 (<module 'atheris' from '/google/src/cloud/aidenhall/py312/google3/./blaze-bin/third_party/py/atheris/str_fuzzing_example.runfiles/google3/third_party/py/atheris/__init_
_.py'>)                                                                                                                                                                                                             
            160 LOAD_ATTR               14 (_trace_branch)                                                                                                                                                          
            180 LOAD_CONST              10 (2)                                                                                                                                                                      
            182 CALL                     1                                                                                                                                                                          
            190 POP_TOP                                                                                                                                                                                             
            192 CACHE                                                                                     
            194 STORE_FAST               4 (x)                                                                                                                                                                                                                                                                                                                                                                                          
 45         196 LOAD_FAST                2 (allstrs)
                                                                                                          
 47         198 LOAD_FAST                4 (x)
 68         200 BINARY_OP               13 (+=)
            204 STORE_FAST               2 (allstrs)                                                      
            206 JUMP_BACKWARD           27 (to 154)

So if we write a for loop that looks like this:

stuff = [lambda x, y: (x, y)]
for i in stuff:
  pass

It prints

<built-in method _trace_branch of PyCapsule object at 0x11207f044fc0> 1

because the lack of a NULL value on the stack, causing the 2nd calling convention from above to be used, in which case the iterable item is called instaed of the actual _trace_branch function. I have a code fix for this, it still fails, but I am not blocked on debuggin (for now).

[mahdoosh1]

i'm confused, where is adjust_arg used?

EDIT: i noticed it, it is used when working with instr.arg

[mahdoosh1]

it is acting weird on my side, it is giving me Segmentation Fault.
i won't debug that i don't understand c or cpp enough