Encoding::CompatibilityError when csv in UTF-16 format

[makarychev]

In PR Use String#start_with? to detect leading special character #12 the starts_with_special_character? implementation was changed to use start_with? which throw an error in case CSV in UTF-16LE format

Encoding::CompatibilityError:
       incompatible character encodings: UTF-16LE and UTF-8

[zvory]

Could you provide an example test case that throws this error? Thank you for noticing this and bringing this to my attention.

…

On Fri, May 12, 2023 at 20:33 Maksym ***@***.***> wrote: In PR Use String#start_with? to detect leading special character #12 <#12> the starts_with_special_character? implementation was changed to use start_with? which throw an error in case CSV in UTF-16LE format Encoding::CompatibilityError: incompatible character encodings: UTF-16LE and UTF-8 — Reply to this email directly, view it on GitHub <#17>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBQJZAVSYNSH6ZGO5QCOILXFY32DANCNFSM6AAAAAAX7RKAUQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[makarychev]

Hey @zvory,
Thank you for the prompt response.
Here is the example

require 'csv'
require 'csv-safe'

class MyCSV
  def self.generate_multiplatform
    CSVSafe.generate(col_sep: "\t") { |csv| yield csv }.encode('utf-16le').prepend("\xFF\xFE".force_encoding('utf-16le'))
  end
end

csv = MyCSV.generate_multiplatform do |c|
  c << ['Вода', 'Water', '💧']
end

CSVSafe.parse(csv, col_sep: "\t", encoding: 'utf-16le')

which throw the error

csv-safe-3.2.1/lib/csv-safe.rb:24:in `start_with?': incompatible character encodings: UTF-16LE and UTF-8 (Encoding::CompatibilityError)
        from ruby-3.2.1/gems/csv-safe-3.2.1/lib/csv-safe.rb:24:in `starts_with_special_character?'
        from ruby-3.2.1/gems/csv-safe-3.2.1/lib/csv-safe.rb:36:in `prefix_if_necessary'
        from ruby-3.2.1/gems/csv-safe-3.2.1/lib/csv-safe.rb:47:in `sanitize_field'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/fields_converter.rb:59:in `block (2 levels) in convert'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/fields_converter.rb:56:in `each'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/fields_converter.rb:56:in `block in convert'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/fields_converter.rb:50:in `collect'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/fields_converter.rb:50:in `with_index'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/fields_converter.rb:50:in `convert'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:1267:in `emit_row'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:1017:in `block in parse_quotable_loose'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:52:in `block in each_line'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:49:in `each_line'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:49:in `each_line'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:963:in `parse_quotable_loose'
        from ruby-3.2.1/lib/ruby/3.2.0/csv/parser.rb:406:in `parse'

[zvory]

Ah unfortunately I'm way busier than I expected and really don't have a moment to spare. I'm going to try to make time for fixing this but I don't know when I'll be able to. If you put up a PR though I'll review it and try to get it merged.