Broken with openssl 3.0

[RaphaelPour]

With OpenSSL 3.0 (which is default since at least Ubuntu 22.04 Jammy), the cipher ECB isn't supported anymore. This will break vncdes#40.

Short reproducer:

> ruby -r openssl -e "p OpenSSL::Cipher::DES.new(:ECB)"
/usr/lib/ruby/3.0.0/openssl/cipher.rb:21:in `initialize': unsupported (OpenSSL::Cipher::CipherError)
	from /usr/lib/ruby/3.0.0/openssl/cipher.rb:21:in `block (3 levels) in <class:Cipher>'
	from -e:1:in `new'
	from -e:1:in `<main>'

Workaround

The current workaround is allowing legacy crypto via openssl.cnf or pin your gem to an older version with selfmade crypto.

[kaspergrubbe]

Interesting, thanks for reporting this.

I am not sure what the correct way forward is, I assume we use that cipher because it is required for some VNC-servers out there, but there might be a newer cipher we will have to use instead, I will have to investigate further.

[deemytch]

I hope that will add a little glue how to test that.
I took error below when try connected to selenoid/vnc_chrome docker container.

[5] pry(main)> Net::VNC.open('localhost:27021', :password => 'selenoid'){|vnc| vnc.key_press :return }
OpenSSL::Cipher::CipherError: unsupported
from /usr/lib/ruby/3.0.0/openssl/cipher.rb:21:in `initialize'

[kaspergrubbe]

This is still on my radar, but I haven't had much time lately to look into it :(

[deemytch]

So, after a day of googling and a little bit researching I found two variants to resolve that.

1. Adding to the very first line lib/cipher/vncdes.rb

ENV['OPENSSL_CONF'] = "#{ __dir__ }/add_legacy.cnf"

and cnf file, that switches on the "legacy" provider.
But this may affect other code, using this gem.
add_legacy_cnf.txt

2. Switching this lib to the latest 3.8 vnc proto and using more modern ciphers.
   I don't know how long will take that work.

[aquasync]

Another option would be to revert d4a41c1, then it won't have any dependency on openssl for the authentication handshake.

[RaphaelPour]

1. Adding to the very first line lib/cipher/vncdes.rb

ENV['OPENSSL_CONF'] = "#{ __dir__ }/add_legacy.cnf"

I guess this would introduce some kind of vulnerability as the env variable is set for all gems an application requires. Gems depending on openssl that need to probe which algorithms are supported might use a legacy one that is not secure anymore.

EDIT

But this may affect other code, using this gem.

Nevermind, you already pointed it out

[kaspergrubbe]

I've released a new version of the gem without DES through OpenSSL, however, if you agree, we could bring back the C-version that was used in the gem previously.

[aquasync]

It isn't an issue for me either way, but just wanted to clarify that the old Cipher::DES class was pure ruby - the .c file in contrib was merely the original code that it was ported from.