Crash on first line of code

[Dead2]

Tested using:
Clang 12.0.1 (Fedora 12.0.1-1.fc34)
Built zlib-ng from develop branch and ran the resulting example (basic self-tests) binary.

export SZ_CODE=1 SZ_LINK=0 SZ_HEAP=0 SZ_STACK=0 SZ_CLEAN=1
cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DWITH_BENCHMARKS=OFF -DWITH_GTEST=OFF -DWITH_OPTIM=OFF -DWITH_NEW_STRATEGIES=OFF -DCMAKE_C_COMPILER=/home/hansr/github/stabilizer/szcc -DCMAKE_C_FLAGS="-Wall -Wpedantic -g3 -ggdb -O0" .`

gdb ./example

Program received signal SIGSEGV, Segmentation fault.
main (argc=1, argv=0x7fffffffde58) at /home/hansr/github/zlib/zlib-ng/test/example.c:960
960         if (zVersion()[0] != myVersion[0]) {
=> 0x00000000004040e4 <main+20>:        ff 15 56 03 70 15       call   *0x15700356(%rip)        # 0x15b04440

This is the first line of actual code in main(), after variable definitions.

Did a little experiment to see whether adding a line above that with __asm__("nop"); would do anything at all.

#0  main (argc=1, argv=0x7fffffffde58) at /home/hansr/github/zlib/zlib-ng/test/example.c:962
962         if (zVersion()[0] != myVersion[0]) {
=> 0x00000000004040e5 <main+21>:        ff 15 55 00 3a a0       call   *-0x5fc5ffab(%rip)        # 0xffffffffa07a4140

Not much changed, but the call address seems to be very random, unless there is a bit pattern I am not seeing.

Seems to me like something is still quite wrong somewhere in the Stabilizer plugin pass when it comes to patching the call addresses.

[magras]

gdb ./example

I strongly recommend to test programs compiled with SZ_CODE=1 without a debugger, because it might interfere with stabilizer runtime.

Stabilizer runtime writes int3 (which raises SIGTRAP) and an address of meta information (such as replaced code, location of the relocated function, etc) to the beginning of all functions registered for relocation before original main is called. The first time function called, the SIGTRAP handler copies the function code to a random address on a heap and replaces int3 with a trampoline to the relocated function. So if debugger intercepts SIGTRAP and does not pass it back to the program, cpu executes a code interleaved with a data. In particular main+24 contains the address of aforementioned meta information allocated on a heap, so it will naturally change from one run to the next.

If stabilizer works correctly you should see the original addresses of a function only before the signal raised (rip points to the first instruction of function which is int3), or during redirect to the relocated function (21 first bytes of a function replaced with the trampolin).

[Dead2]

Hmm, valid point.
That limits my ability to debug things by a lot though, as my only (admittedly limited) experience with debugging on this level is through gdb.

That said, example also crashes without compiling with -g3 -ggdb and not running under gdb, also without Z_TLS enabled. But I have no idea whether it crashes in the same way as under gdb.

[magras]

I use coredumps and printfs. They are slower, but at work I rarely had a luxury of interactive debugging. With stabilizer I find it very useful to log relocation addresses, so that I can match raw asm to symbols.

I'll take a look at Z_TLS. I assume it is a flag, that enables usage of TLS in zlib-ng, right?

[magras]

I don't see any crashes with Z_TLS disabled:

root@8dec1434f819:/usr/local/src/zlib-ng# cmake -B build -DCMAKE_VERBOSE_MAKEFILE=ON -DWITH_BENCHMARKS=OFF -DWITH_GTEST=OFF -DWITH_OPTIM=OFF -DWITH_NEW_STRATEGIES=OFF -DCMAKE_C_COMPILER=szcc -DCMAKE_C_FLAGS="-Wall -Wpedantic -g3 -ggdb -O0 -DZ_TLS=" . --fresh
...
root@8dec1434f819:/usr/local/src/zlib-ng# cmake --build build --clean-first
...
root@8dec1434f819:/usr/local/src/zlib-ng# build/example
 [libstabilizer.cpp:57] Initializing Stabilizer
 [libstabilizer.cpp:60] Stack top is at 0x7fff80f4c280
 [libstabilizer.cpp:68] Signal handlers installed
 [libstabilizer.cpp:76] Trapped all functions
 [libstabilizer.cpp:80] Set re-randomization timer
 [libstabilizer.cpp:87] Finished with program constructors
zlib-ng version 2.1.0.devel = 0x02010000, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
gzgets(): hello, hello!
inflate(): hello, hello!
large_inflate(): OK
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
deflateBound(): OK
deflate_copy(): OK
deflateGetDictionary(): hello, hello!
deflateSetHeader(): OK
deflateTune(): OK
deflatePending(): OK
deflatePrime(): OK
 [libstabilizer.cpp:95] Shutting down

[Dead2]

That is quite strange.
Only thing I can think of is that that LLVM 14 works but LLVM 12 does not. I think I will have to try to get LLVM 14 installed.
(I do clean builds of course, I just didn't include that extra fluff in the above post)

Regarding debugging; Printf I am used to of course, but I have never tried to do any segfault debugging with that, should work but it does sound a bit cumbersome.
Coredumps is of course a great idea, I have only really used that to debug intermittent crashes on production servers before. My mind just discarded that option automatically when you said not to use a debugger, but of course you meant not to connect a debugger to the process. 😆

[magras]

If you use containers, I have a couple for testing stabilizer.

This one is for stabilizer and has customizable llvm version (unfortunately llvm provides packages for debian:unstable starting from version 13):

FROM docker.io/debian:unstable

RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get upgrade -y

# llvm

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get install -y --no-install-recommends lsb-release wget software-properties-common gnupg

ARG LLVM_VERSION
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" llvm.sh $LLVM_VERSION all

# stabilizer

RUN ln -sT /usr/bin/clang-$LLVM_VERSION /usr/bin/clang && \
    ln -sT /usr/bin/clang++-$LLVM_VERSION /usr/bin/clang++ && \
    ln -sT /usr/bin/llvm-link-$LLVM_VERSION /usr/bin/llvm-link && \
    ln -sT /usr/bin/llvm-as-$LLVM_VERSION /usr/bin/llvm-as && \
    ln -sT /usr/bin/opt-$LLVM_VERSION /usr/bin/opt && \
    ln -sT /usr/bin/llc-$LLVM_VERSION /usr/bin/llc && \
    ln -sT /usr/bin/lld-$LLVM_VERSION /usr/bin/lld

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get install -y --no-install-recommends make python3-distutils

COPY . /usr/local/src/stabilizer
WORKDIR /usr/local/src/stabilizer

RUN CC=clang CXX=clang++ INCLUDE_DIRS="/usr/include/llvm-$LLVM_VERSION /usr/include/llvm-c-$LLVM_VERSION" make clean all
RUN ln -s /usr/local/src/stabilizer/szcc /usr/local/bin && \
    ln -s /usr/local/src/stabilizer/szcc++ /usr/local/bin && \
    ln -s /usr/local/src/stabilizer/LLVMStabilizer.so /usr/local/bin && \
    ln -s /usr/local/src/stabilizer/libstabilizer.so /usr/lib

This one for zlib-ng:

FROM stabilizer

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get install -y --no-install-recommends cmake git ninja-build

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get install -y --no-install-recommends gdb neovim

COPY . /usr/local/src/zlib-ng
WORKDIR /usr/local/src/zlib-ng

ENV SZ_CODE=1 SZ_LOWER=1
RUN cmake -B build -G Ninja -DCMAKE_C_COMPILER=szcc -DCMAKE_CXX_COMPILER=szcc++ -DCMAKE_BUILD_TYPE=Debug . --fresh
RUN cmake --build build