Skip to content

8bit quantization  #8

@CornN64

Description

@CornN64

To improve the 8bit quantization in the RTL mode with ½ LSB and round it to the nearest LSB I would suggest to replace the lines

*(data++) = (unsigned char)(((*xi << sample_shift) >> 8) + 128);
*(data++) = (unsigned char)(((*xq << sample_shift) >> 8) + 128);

with

*(data++) = (unsigned char)((((*xi << sample_shift) + 0x80) >> 8) + 128);
*(data++) = (unsigned char)((((*xq << sample_shift) + 0x80) >> 8) + 128);

or even more optimized

*(data++) = (unsigned char)(((*xi << sample_shift) + 0x8080) >> 8);
*(data++) = (unsigned char)(((*xq << sample_shift) + 0x8080) >> 8);

Cheers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions