I was pulling my hair out trying to figure out why my stream looked terrible, and it was because jellycon decided to use transcoding (for unknown reason). Then I found the "force transcode max bitrate" setting, turned it up to "maximum" and that fixed the poor quality.
I don't really understand why this is even enabled by default. Shouldn't the client just try to transcode as quickly as it can by default?