Q:

Python: solving unicode hell with unidecode

I have been working on ways to flatten text into ascii. So ā -> a and ñ -> n, etc.
unidecode has been fantastic for this.
# -*- coding: utf-8 -*-
from unidecode import unidecode
print(unidecode(u"ā, ī, ū, ś, ñ"))
print(unidecode(u"Estado de São Paulo"))

Produces:
a, i, u, s, n
Estado de Sao Paulo

However, I can't duplicate this result with data from an input file.
Content of test.txt file:
ā, ī, ū, ś, ñ
Estado de São Paulo

# -*- coding: utf-8 -*-
from unidecode import unidecode
with open("test.txt", 'r') as inf:
    for line in inf:
        print unidecode(line.strip())

Produces:
A, A<<, A<<, A, A+-
Estado de SAPSo Paulo

And:

RuntimeWarning: Argument  is not an unicode object.
  Passing an encoded string will likely have unexpected results.

Question: How can I read these lines in as unicode so that I can pass them to unidecode?

A:

Use codecs.open
with codecs.open("test.txt", 'r', 'utf-8') as inf:

A:

import codecs
with codecs.open('test.txt', encoding='whicheveronethefilewasencodedwith') as f:
    ...

The codecs module provides a function to open files with automatic Unicode encoding/decoding, among other things.

