How to get the coverage of Unicorn

[jimmysitu]

Hi, All

I am following the README file to run silifuzz, and got some confused in Prework (fuzzing Unicorn target)

It seems every unicorn file add coveage option with COV_FLAGS_FILE, but I run the command

"${SILIFUZZ_BIN_DIR}/external/com_google_fuzztest/centipede/centipede" \
  --binary="${SILIFUZZ_BIN_DIR}/proxies/unicorn_x86_64" \
  --workdir=/tmp/wd \
  --j=4 --num_runs=100

I only get the corpus in /tmp/wd, where can I found the coverage info for these runs?

BTW, how about the coverage info of pre-generated corpus mentioned in here
Thanks

[ncbray]

What is your goal / why do you want the coverage information?

Background:

Every time centipede finds an "interesting" input (new or relatively rare coverage points) it writes the input (but not coverage) into ${workdir}/corpus.${shard_num}, where each fuzzing thread gets a different shard_num.

Centipede also persists the coverage data, but how is a bit more complicated. Centipede has a number of different notions of "coverage" (also known as "features") but for now let's only talk about pc/program counter/line coverage.
https://clang.llvm.org/docs/SanitizerCoverage.html
Taking a step back, the number and meaning of each coverage point depends on the exact binary being fuzzed. If the binary is recompiled, the meaning of each coverage point can change - maybe because the code has changed, maybe because the compiler is doing something different. Because of this, Centipede stores the coverage separately from the inputs - in ${workdir}/${binary}-${binary_hash}/features.${shard_num}. On startup, if Centipede detects that the binary name or hash is different than before, it will create a new directory to hold the features. Centipede will then load every input from the corpus. If Centipede cannot find cached feature data for a particular input (if the binary hash has changed or if the feature directory was deleted or does not exist) it will re-run the input and generate fresh coverage data with the latest binary. This has a small to moderate performance impact, so it is normally cached.

This is why the pre-generated corpus contains no coverage into - Centipede will immediately regenerate it, and it really depends on what binary you are using.

So far, everything I've talked about is a notion of "coverage" that's entirely internal to centipede - it doesn't help you much, as a human being. If you want to see a human readable line coverage, run centipede with --telemetry_frequency=1, or some other non-zero value. This will write ${workdir}/coverage-report-${binary}.000000.{initial,latest,final}.txt. This will also slow down centipede, so you probably don't want to pass this flag if you are running for a long time.

Note that line coverage is just the start, Silifuzz proxies can also generate other types of coverage such as "this particular bit in the register file was flipped from 0=>1 by the input".
Start digging into the code starting here if you want to know more: https://github.com/google/silifuzz/blob/main/proxies/unicorn_x86_64.cc#L89
Centipede also has other sorts of features such as --pc_pairs, etc.

Please ask if you have any more questions.

[jimmysitu]

Hi, @ncbray

Thanks for the explanation.

My goal to check the coverage info is try to understand the coverage rate vs run time.

I suppose it is very time consuming to get new features cover when coverage rate reaches high.
Is there any coverage rate can be calculated with features in ${workdir}/${binary}-${binary_hash}/features.${shard_num}?
Does Centipede know the total number of features for a given binary?

Thanks

[ncbray]

Have you looked at Centipede's console output?

ft: will be the total number of features, from all domains. This metric is not entirely useful because different domains have different numbers of features, potentially my several orders of magnitude.
cov: is traditional PC coverage
cmp: is traditional "compare" coverage
usr0-usr7: are directly generated by the proxy.
See: https://github.com/google/silifuzz/blob/main/proxies/arch_feature_generator.h#L80
usr0: is the number of distinct opcodes found.
usr1: the number of distinct ordered pairs of opcodes executed one after another (add, mul) != (mul, add)
etc.

The "total" number of features is not particularly meaningful, unfortunately. The proxy may have dead code, either directly (it supports other CPU models that aren't being fuzzed) or indirectly (Silifuzz filters out syscalls, non-deterministic instructions, etc, so they'll never be covered). We tend to consider the "maximum" coverage to be the best we've seen previously.

In practice on a 96-core machine I can saturate most of the coverage domains in 24-48 hours, IIRC.

The features.* files are really only meant for Centipede. The also don't have a sense of time, so "rate" would be difficult to compute. You could extract time / fuzzing iteration from the log lines.

What are you trying to figure out by measuring the coverage?

[jimmysitu]

Hi, @ncbray

I got your point, one of our research topic is to improve the fuzzing efficiency. and we are try to figure out if there is any chance to speed up centipede mutator reach the saturate coverage.

Any suggestion?

Thanks

[ncbray]

Are you looking at Silifuzz+Centipede specifically, or Centipede in general? Silifuzz is a bit weird, as far as "fuzzing" goes, so results on one side of the fence may not map to the other.

I've experimented with a domain-specific mutator:
https://github.com/google/silifuzz/blob/main/fuzzer/BUILD
The goal was to increase the number of branches in the random tests because most randomly generated branches will land outside the test. To this end is succeeded, as well as finding the coverage a bit faster, but at a small loss of PC coverage (I didn't get a chance to dig in and see exactly what the skew was - it may have been as simple as the new mutator never jumping into the middle of a an instruction and that avoided some cache clearing. Missing coverage is not always bad. Or it may have been something more significant.) When I deployed the tests on known defective machines, however, the hit rate did not really change. That is ultimate measure we care about - finding CPU defects - but it's difficult / time consuming / statistically fraught to actually measure. Coverage is an approximation of what we actually care about. We know the coverage metrics we have do not fully cover the full space of possible CPU defects.

[jimmysitu]

@ncbray ,
Yes, I am look at Silifuzz+Centipede specifically. Thanks for your explanation.
Base on your experience, how do you know one proxy should be retired, and hired a new one?