Q:

ruby: fix mostly-utf8 strings

I have input data (via File.popen) that is mostly utf8, but occasionally there are iso8859-1 characters in them. I want everything that is not a valid utf8 sequence interpreted as iso8859-1 and replaced with the corresponding (two-byte) utf-8 sequence (and the result as an UTF-8 encoded string).
What is an efficient way to do this in ruby? This will treat an entire git log output, so it should be reasonably fast.

A:

since ruby 2.1.0 (afaik) you can use scrub to do this kind of ugly encoding stuff:
https://ruby-doc.org/core-2.1.0/String.html#method-i-scrub

If the string is invalid byte sequence then replace invalid bytes with
  given replacement character, else returns self. If block is given,
  replace invalid bytes with returned value of the block.

"abc\u3042\x81".scrub #=> "abc\u3042\uFFFD"
"abc\u3042\x81".scrub("*") #=> "abc\u3042*"
"abc\u3042\xE3\x80".scrub{|bytes| '<'+bytes.unpack('H*')[0]+'>' } #=> "abc\u3042<e380>"

