Test Data Randomly Overwritten

[taylorflatt]

Description

When running the partial_write_single_test script, after the single cycle of clients has completed (possibly successfully), an old .output file gets overwritten or emptied to 0 bytes. This behavior would likely extend to the partial_write_cycle_test script as well since they are similarly coded.

Expected Behavior

I expect this to never happen. There doesn't seem to be code that would explain this behavior in the scripts. I don't expect a test's output to be overwritten once it has concluded. It is theoretically possible that NONE of the data gets written, hence the zero. However, this doesn't seem to be the case if the directory is watched while the tests run.

Actual Behavior

A prior test gets overwritten or emptied so the size goes from, let's say, the correct size of 1000 to 0.

Steps to Reproduce

It can be difficult to reproduce. But a tried and true method is to run the script repeatedly in one terminal and a watch -dp ls -la in another terminal to watch for the behavior to occur.

Your Environment

• Compiler: gcc 4.9.2
• OS (and version): Debian 8.9 (Jessie)
• RAM: 128Gb
• CPU (and cores): Intel dual Xeons (32 cores)

Solution/Suggestion

I'm not sure as to what could be a positive direction to solve this issue at the moment.

[taylorflatt]

I created a quick and easy script for reproduction of the bug:

#!/bin/bash

YELLOW=`tput setaf 3`
END_COLOR=`tput sgr0`

cycle=0

declare -a file_list
file_list+=("500.output")
file_list+=("1000.output")
file_list+=("2000.output")
file_list+=("3000.output")
file_list+=("4000.output")
file_list+=("10000.output")
file_list+=("20000.output")
file_list+=("100000.output")

while true; do
    ((cycle++))
    echo -e "\n${YELLOW}Starting cycle $cycle...${END_COLOR}\n"
    # Check if any of the output files are zero.
    for (( i=0; i<${#file_list[@]}; i++ )); do
        if [[ ! -s ${file_list[$i]} ]]; then
            echo "${file_list[$i]} is ZERO! Failure on cycle $cycle."
            exit 1
        fi
    done

    # Run the script again.
    ./partial_write_single_test.sh
done

exit 0

Make sure the output files have been created once prior to running the script. This can be done by running partial_write_single_test.sh once prior to the debug script.

Also, I suspect the overwrite might be occurring as the client running the tests exits (which might happen after another test has started).