Bug: Error Occurs When Running webFuzz.py

[DMIND-NLL]

Description

I encountered an error while attempting to reproduce the steps outlined in the Example.md documentation for webFuzz. Specifically, the issue arises during Step 2: "Run the proxy to get JavaScript generated URLs". After interacting with the browser by clicking several features and subsequently closing the browser window, the command line remains active for over an hour without completing and displays error messages.

Command Executed：

~/webFuzz/env/bin/python /home/ubuntu/webFuzz/webFuzz/webFuzz.py -vv --driver /usr/local/bin/geckodriver -m /var/www/html_instrumented/instr.meta -w 8 -b 'wp-login|action|logout|' -p -s -r simple http://192.168.1.200

Error Message：

webFuzz
-----

Stats

Runtime: 0.02 min
Total Requests: 9
Throughput: 0.00 requests/s
Crawler Pending URLs: 0





--- Logging error ---
Traceback (most recent call last):
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 329, in _flatten
    result = self._flattened[id(obj)]
KeyError: 140105777691808

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit
    msg = self.format(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 929, in format
    return fmt.format(record)
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/types.py", line 322, in format
    return formatter.format(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 668, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage
    msg = msg % self.args
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/node.py", line 343, in __str__
    return self.json
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/node.py", line 198, in json
    self._json = jsonpickle.encode(self, unpicklable=False)
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 166, in encode
    context.flatten(value, reset=reset), indent=indent, separators=separators
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 366, in flatten
    return self._flatten(obj)
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 331, in _flatten
    result = self._flattened[id(obj)] = self._flatten_impl(obj)
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 386, in _flatten_impl
    return self._pop(self._flatten_obj(obj))
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 419, in _flatten_obj
    raise e
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 413, in _flatten_obj
    return flatten_func(obj)
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 728, in _ref_obj_instance
    return self._flatten_obj_instance(obj)
  File "/home/ubuntu/webFuzz/env/lib/python3.8/site-packages/jsonpickle/pickler.py", line 660, in _flatten_obj_instance
    state = obj.__getstate__()
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/node.py", line 333, in __getstate__
    state['cover_score'] = str(f"{self.cover_score:.3f}")
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/node.py", line 175, in cover_score
    return 100 * score / count
ZeroDivisionError: division by zero
Call stack:
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz.py", line 13, in <module>
    fuzzer.run()
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/fuzzer.py", line 190, in run
    return asyncio.run(self.async_run(interface))
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/worker.py", line 159, in run_worker
    for (src, new_request) in iter_join(primary=self._crawler,
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/misc.py", line 63, in iter_join
    yield (primary, next(primary))
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/crawler.py", line 221, in __next__
    if self._should_block(new_request):
  File "/home/ubuntu/webFuzz/webFuzz/webFuzz/crawler.py", line 127, in _should_block
    logger.info("Blocked %s", new_request)
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.

Environment Information

Operating System: Ubuntu 20.04
Python Version: Python 3.8.10
webFuzz Version: v1.2.3
geckodriver Version: v0.35.0
Browser Version: Mozilla Firefox 131.0
Web Application: WordPress 6.1.1

I would greatly appreciate it if you could help me investigate this issue. Thank you for your time and assistance!

[fqwadad]

Hello, when I execute the second step, python3 webFuzz.py -w 8 --meta /var/www/html/wordpress_instrumented/instr.meta --driver /snap/bin/geckodriver -vv -p -r simple http://localhost/wordpress/wp-admin/index.php, there is a problem selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status 1. Have you encountered this? Thank you
FBB534F283D169A404554F94695EFE3B

61790A35E11CC3A5F3AA7A4138E9787A

[DMIND-NLL]

@fqwadad
Certainly, I have also encountered this error. My solution was to avoid running the command as root, which effectively resolved the issue you mentioned. You might try switching to a regular user when running the command, and I hope this helps you. If your problem is resolved, could you check whether the issue I raised appears?

[fqwadad]

Ok, thanks for your answer, I will try it and see if your problem occurs.

[fqwadad]

截图 2024-10-15 17-12-33
Thanks for your answer! I solved the problem of the second step, and when I run the third step, I don't encounter your problem, but the coverage is 0.
python3 webFuzz.py --ignore_4xx -w 8 --meta /var/www/html/wordpress_instrumented/instr.meta --driver /usr/local/bin/geckodriver -b 'wp-login.php|action|logout|*' -vv --request_timeout 100 --seed_file ./seeds/webFuzz_seed_15-10_16:5.json -s -r simple

[DMIND-NLL]

@fqwadad

I did not encounter a situation where the coverage is 0, but when I perform the third step, I am always stuck on the login interface. The browser repeatedly pops up and asks me to log in. Even if I successfully log in and close the browser, webFuzz seems unable to retrieve the login credentials....

python webFuzz.py --ignore_4xx -w 8 --meta /var/www/html/wordpress_instrumented/instr.meta --driver /usr/local/bin/geckodriver  -b 'wp-login.php|action|logout|*' -vv --request_timeout 100 --seed_file seeds/webFuzz_seed_16-10_10\:6.json -s  --catch_phrase Howdy -r simple http://192.168.1.200/wp-admin/index.php

Additionally, when I perform the second step, it keeps executing the same few links, and the coverage remains at the same value.

python webFuzz.py -vv --driver /usr/local/bin/geckodriver --meta /var/www/html/wordpress_instrumented/instr.meta -w 8  -p  -r simple http://192.168.1.200/wp-admin/index.php

[fqwadad]

Thank you, I have solved the problem of 0 coverage. As for your problem, I did not encounter it. 1. Maybe it is a Firefox version problem? https://firefox-source-docs.mozilla.org/testing/geckodriver/Usage.html#Running-Firefox-in-an-container-based-package. 2. Check that you use the correct java version, does browsermob-proxy run? webFuzz/drivers/browsermob-proxy-2.1.4/bin/./browsermob-proxy?

[fqwadad]

Hello, may I ask if your experiment was successful? When I ran it, the coverage was very low. Is this related to my seeds? If you could answer, I would be extremely grateful.

[DMIND-NLL]

@fqwadad
I also couldn't successfully reproduce the experiment. I’ve tried running it for over 750 minutes, but the coverage is still low, and it seems to keep testing the same few links back and forth.

[ovanr]

@fqwadad I also couldn't successfully reproduce the experiment. I’ve tried running it for over 750 minutes, but the coverage is still low, and it seems to keep testing the same few links back and forth.

How many URLs that are not JavaScript files does your seed file have (step 2 of Example.md)? Does webFuzz fuzz these URLs at the beginning of the session?

[fqwadad]

In my seed file, there are over twenty non JavaScript URLs. When creating the seed file, I performed some operations such as adding, deleting, modifying, and querying. Is there a problem with this? How did you create the seed file? If you could answer, I would greatly appreciate it.

[ovanr]

Does your seed file contain all the completed fields of each submitted form? Make sure that you exercise Wordpress as much as possible and that the form submissions (e.g. creating a post) succeed.
Also at the beginning of the fuzzing session, does webFuzz say that it tries these forms?

[fqwadad]

Sorry, webfuzz did not execute these forms but instead repeated testing some of the same URLs, such as http://localhost/wordpress/wp-admin/edit.php , http://localhost/wordpress/wp-admin/post.php , http://localhost/wordpress/wp-admin/plugin-install.php etc.

[ovanr]

From what I can see in the seed file, it seems that your web server serves both the instrumented and non-instrumented version of wordpress (prefix wordpress and wordpress_instrumented).
I imagine the URLs that start with wordpress are not instrumented so webFuzz doesn't get any coverage feedback.
The web server should serve only the instrumented version