Byte error in transmission using "responses::FileContentResponse"

[squonk11]

Currently I am transmitting an XML file (size approx. 700kB) via responses::FileContentResponse from SDcard to the Browser using a REST request/response. In the Browser I process this file using DOMParser()).parseFromString(str, "text/xml"). Approx. every 3rd transmission I get an error message from this parser saying:

XML Parsing Error: not well-formed Location: http://localhost:8080/test Line Number 2977, Column 16:

I analyzed the wireshark trace on some malicious transmissions and found out that on arbitrary positions in this file a character is corrupted; example: in the word "protocol" the last "o" (hex: 0x6F) is being replaced by 0 (hex: 0x00).
Now I am asking myself where this problem comes from. Currently I see the following possibilities:

1. Error in reading from the SD card
   The error might arise while reading the file from the SD card. Until now I did already many activities with reading files from the SD card and I never saw such an issue. So I think this root cause is not very likely.
2. Error in processing of the data stream in Smooth
   I guess the responses::FileContentResponse is also being used when transmitting plain HTML pages. Since simple HTML pages seem to work quite well (until now I never saw a problem here), I think also this root cause is quite unlikely the reason.
3. Error in lwip
   Since lwip is very widely used I don't consider any specific problem here.
4. Error during TCP/IP transmission in my network
   AFAIK TCP/IP tranmission has integrated crc checks. Additionally I don't see any other issues in my local network. So I guess this also is not the problem.

Having these possible root causes in mind I currently don't know which one is more likely. Or do you see other possible root causes which I did not consider until now? Do you have a suggestion where I should investigate first?

[squonk11]

As a first step I checked the data being read from the SDcard by implementing an analysis of read data; here I simply counted the zeros read from the file (knowing that the XML file does not have any zero). With this modification I have the following observations:

1. With this analysis included the error happens more seldom; maybe every 10th transmission of the file
2. When the error occurs, I see always one arbitrary byte flipped to 0 (0x00) in the wireshark trace but the number of counted zeros when reading the file is always zero.
   This lets me draw the following conclusions:

• The file on the SDcard and reading from the SDcard is not the problem
• since the error occurs more seldom with the analysis function included it seems to be some kind of timing issue?

[squonk11]

Since I had the suspicion that it is somehow related to a timing problem (see previous post), I modified the timeout for the select() call. I had it set to 0ms since some time (see this thread). When increase this timeout to 10ms the result is still very similar: the problem still occurs but maybe a little bit more seldom.

[PerMalmberg]

Oh-boy, this could be anything. I agree that lwip is likely not to blame, and probably not the SDCard either.

The only way to find the cause is to do what you've already begun - a systematic trial and error.

I'd setup a test on the host with a HTTP server that generates (i.e. purely in-memory) a sequence of bytes of different lengths (from small to large) that can be verified on the receiving end. Each sequence should preferably be reproducible such that the entire test can be repeated.

If I don't remember wrong, there's something among the examples that does something close to this.

[squonk11]

In the examples there is a test with multiple file uploads (from PC to ESP32). But here the problem occurs in the opposite direction (from ESP32 to PC). Currently I encounter this problem not so often - maybe every 10th or 20th transmission of a large file. For the time being I can live with it but later I definitely need to fix it.