Q:

Splitting a large libsvm formatted file with command line tools

I have a very large libsvm formatted file (150GB) where each line is like the following:
-1 430018:1 429765:1 428103:1 428954:1 430172:1 427300:1 429485:1 432367:1 427059:1 426870:1 426556:2
(first token is label, and the others are feature:value pairs).
I wrote a program in Python to convert this into [label, [array of feature ids], [array of values]] such as:
[-1, [430018,429765,428103,...  ], [1,1,1,...] ] 
but it runs extremely slow in Python (I was able to process only 10GB file in 3 hours).
Is there a way to take this file and change the format to [label, [array of feature ids], [array of values]] per line and write it to another file?
For the reference, here is the Python script I wrote to convert each line:
def convert(f, line)
        l = line.strip().split(" ")
        label = int(l[0])
        x = [map(int, f.split(":")) for f in l[1:]]
        x_idx, x_val = zip(*x)
        f.write(str([label, x_idx, x_val])+"\n")

A:

sed -u --posix "
s/^\([^ ]*\) \{1,\}\(.*\)$/[\1,[,\2 ][/
:peer
s/,\([[:digit:]]\{1,\}\):\([[:digit:]]\{1,\}\) \(.*\)$/,\1,\3,\2/
t peer
s/\[,/[/g
s/,]/]/g
s/$/]/
" yourfile

The -u is mandatory on this kind of HUGE file. --posix is for GNU sed
-i if direct modification, if not redirect it.
I have no idea of performance on such a file. if too slow, will can still try other sed more efficient in performance but maybe harder to define.

