Q:

Sed alternative for search and replace on very long lines

I have files that were generated by a program that did not put newlines at the end of records. I want to put newlines between the records, and I can do so with a simple sed script:
sed -e 's/}{/}\n{/g'

The problem is that the input files are multiple gigabytes in size, and therefore the input lines to sed are multiple GBs in length. sed tries to hold a line in memory, which doesn't work in this case. I tried the --unbuffered option, but that just seemed to make it slower and did not allow it to finish correctly.

A:

You can use another tool that lets you set the input record separator. For example

Perl
perl -pe 'BEGIN{ $/="}{" } s/}{/}\n{/g' file

The special variable $/ is the input record separator. Setting it to }{ defines lines as ending in }{. That way you can achieve what you want without reading the entire thing into memory.
mawk or gawk
awk -v RS="}{" -vORS= 'NR > 1 {print "}\n{"}; {print}' file 

This is the same idea. RS="}{" sets the record separator to }{ and then you print }, a newline, { (except for the first record) and the current record.

A:

Perl to the rescue:
perl -i~ -e ' $/ = \1024;
              while (<>) {
                  print "\n" if $closing and /^{/;
                  undef $closing;
                  s/}{/}\n{/g;
                  print;
                  $closing = 1 if /}$/;
              } ' input1 input2

Setting $/ to \1024 will read the file in chunks of 1024 bytes. The $closing variable handles the case when a chunk ends in } and the next one starts with {.

