First 5 bytes are "zeroed" out

[ryanjpearson]

Hi, I set up deflate.py with a 32768 OBSIZE and COMPRESS = False. I wrote C code based on test_deflate.py. When it runs, the first 5 bytes that are output on o_byte are zeroed out, but after that, the bytes look correct. The bytes are not coming out shifted. Byte 5 is what byte 5 is supposed to be, when I compared the output with the output of a software inflate function. The code is attached, thanks for any suggestions! (I'm particular wondering if there's an issue in InitDecompress, WriteDecompress, ReadDecompress, or DeflateClockStrobe functions.)

decompress.txt

[tomtor]

@ryanjpearson I will have a look later this week...

[tomtor]

The PNG code makes it a bit harder to find the root cause.

Could you first change your example C code to test the decompression of (EDIT):

static char ctest[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80\x66\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1c\x62\x82\xd3\x00\x35H\rT\x83\xd5\x80\x35h\r\\\x83\xb7|\xbc\x0fjCzP";

which should result in:

"   Hello World! 0         Hello World! 1         Hello World! 2         Hello World! 3         Hello World! 4         Hello World! 5         Hello World! 6         Hello World! 7         Hello World! 8         Hello World! 9         Hello World! 10         Hello World! 11         Hello World! 12         Hello World! 13         Hello World! 14         Hello World! 15         Hello World! 16         Hello World! 17         Hello World! 18         Hello World! 19         Hello World! 20         H"

[ryanjpearson]

Here's a simpler implemenation using the test case you provided. Thanks!

decompress.txt

[tomtor]

Here's a simpler implemenation using the test case you provided. Thanks!

I assume you verified that the first 5 bytes are still damaged (0) and the rest of the output is ok?

[tomtor]

Regarding the test data, why don't you just use (EDIT corrected wrong escapes):

static unsigned char Compressed[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80\x66\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1c\x62\x82\xd3\x00\x35H\rT\x83\xd5\x80\x35h\r\\\x83\xb7|\xbc\x0fjCzP";

In InitDecompress

DeflateWrite(REG_DEFLATE_RESET, 1);
DeflateWrite(REG_I_CLK, 0);
DeflateWrite(REG_DEFLATE_RESET, 0);
DeflateWrite(REG_I_CLK, 1);

Has the clock edge reversed, try changing this to:

DeflateWrite(REG_I_CLK, 0); 
DeflateWrite(REG_DEFLATE_RESET, 1);
DeflateWrite(REG_I_CLK, 1); // Low->High transition to reset
DeflateWrite(REG_DEFLATE_RESET, 0);

Let me know if that makes a difference. If not I'll have a look at the rest of the code.

Could you also tell me a bit more of your use case? Why are you driving/testing the decompress logic from C code?

That will be very slow, just using the processor and a software implementation would have the same or higher speed.

test_deflate_bench in test_deflate.py is an example driving the logic directly.

[ryanjpearson]

Thanks for the suggestion! I'm away from the hardware for 2 weeks, but I will give it a shot then.

Isn't the string you provided an ascii representation of the data? I used binascii.b2a_hex in python to convert it to the bytes in the code. I wasn't sure of any C functions that would convert "/x95" to just a char with value of 0x95. I'm new to that notation, and I thought it was python specific.

I wasn't certain the micro-controller I'm using would have the memory needed for inflate. I may move more functionality to the FPGA, but I first wanted to verify I was able to interface to the inflate logic correctly.

[tomtor]

There is indeed an issue with 4 (3 reported by the compiler because it exceeds the 0-255 range) of the encoded bytes:

tom@swan-ubu20:~/src/HDL-deflate$ cat test.c 
#include <stdio.h>

int main()
{
  static unsigned char Compressed[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80f\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1cb\x82\xd3\x005H\rT\x83\xd5\x805h\r\\\x83\xb7|\xbc\x0fjCzP";
  for (int i= 0; i < sizeof(Compressed)-1; i++)
	  printf("%02x ", Compressed[i]);
  printf("\n");
}
tom@swan-ubu20:~/src/HDL-deflate$ clang test.c; 
test.c:5:109: error: hex escape sequence out of range
  static unsigned char Compressed[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80f\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1cb\x82\xd3\x005H\rT\x83\xd5\x805h\r\\\x83\xb7|\xbc\x0fjCzP";
                                                                                                            ^~~~~
test.c:5:250: error: hex escape sequence out of range
  static unsigned char Compressed[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80f\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1cb\x82\xd3\x005H\rT\x83\xd5\x805h\r\\\x83\xb7|\xbc\x0fjCzP";
                                                                                                                                                                                                                                                         ^~~~~
test.c:5:280: error: hex escape sequence out of range
  static unsigned char Compressed[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80f\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1cb\x82\xd3\x005H\rT\x83\xd5\x805h\r\\\x83\xb7|\xbc\x0fjCzP";
                                                                                                                                                                                                                                                                                       ^~~~~
3 errors generated.

eg \x80f should have been encoded \x80\x66 because f is a hex digit. This is different from Python encoding, where the encoding is limited to 2 hex characters \xHH.

Corrected:

tom@swan-ubu20:~/src/HDL-deflate$ cat test.c 
#include <stdio.h>

int main()
{
  static unsigned char Compressed[] = "\x18\x95u\xcf\xbb\t\x80@\x00\x04\xd1V\xd6\x0e\xdc\xf3\xdf\x81\x1d\xd8\x80\x66\x07\x07\xf6\x1f\x18\t\n;\x13N\xf6$\xedW\xadMG\xbb\xeb\xd9\xa9\xd7\xdbo;\xef\x92\xf7\x90\xf7\x98\xf7\x94\xf7\x9c\xf7\x92\xf7\x9a\xf7\x06\x1c\x62\x82\xd3\x00\x35H\rT\x83\xd5\x80\x35h\r\\\x83\xb7|\xbc\x0fjCzP";
  for (int i= 0; i < sizeof(Compressed)-1; i++)
	  printf("%02x%c", Compressed[i], (i+1)%16 ? ' ': '\n');
  printf("\n");
}
tom@swan-ubu20:~/src/HDL-deflate$ clang test.c; ./a.out 
18 95 75 cf bb 09 80 40 00 04 d1 56 d6 0e dc f3
df 81 1d d8 80 66 07 07 f6 1f 18 09 0a 3b 13 4e
f6 24 ed 57 ad 4d 47 bb eb d9 a9 d7 db 6f 3b ef
92 f7 90 f7 98 f7 94 f7 9c f7 92 f7 9a f7 06 1c
62 82 d3 00 35 48 0d 54 83 d5 80 35 68 0d 5c 83
b7 7c bc 0f 6a 43 7a 50 

[ryanjpearson]

I used your suggestion and fixed a couple of mistakes in the decompress.txt. Attached is the latest.
decompress.txt

The output has bytes 0 through 18, 26, and 40 "zeroed" out. The rest of the bytes are what they should be.

Here's a screen shot of the first 46 bytes of output

[tomtor]

The output has bytes 0 through 18, 26, and 40 "zeroed" out. The rest of the bytes are what they should be.

What happens if you do 2 clock ticks around line 100:

if (DecompressedReadCount < GetOProgress()) {
		did_read = 1;
		DeflateWrite(REG_I_MODE, READ);
		DeflateWrite(REG_I_RADDR, DecompressedReadCount);
		DeflateClockStrobe(); DeflateClockStrobe(); // TWO TICKS
		DecompressedReadCount++;
	}

[ryanjpearson]

That made the problem go away! Thank you!

[tomtor]

That made the problem go away! Thank you!

That's good news! You will probably only need the extra tick when

DecompressedReadCount + 1 == GetOProgress()

[ryanjpearson]

Now, I'm trying to push through a deflate block of size 705 bytes. o_iprogress is stalling at 481, meaning I'm only able to write 513 bytes. This number is interesting (481 = 512 - 32 + 1). Any suggestions? Thanks!

[tomtor]

The standard input buffer is 512 bytes and cyclic. 32 bytes are reserved for look back when compressing (could also apply when decompressing).

So if you are feeding data to decompress and it does not fit completely in the input buffer, you should also start the decompression and read the result so that the input buffer is emptied and you can reuse it in a cyclic way (after writing slot 511, continue with slot 0).

See the test code around:

HDL-deflate/test_deflate.py

Line 159 in 9a561a6

if o_iprogress > i - MAXW: