
Ask HN: AI Sign Language translators - zunzun
Can AI be made to translate between different sign language variants? I would think computerized avatars could easily be trained, and grant money for such projects should be easy to come by. Computerized vision should be able to read sign language - especially in specific, limited visual settings and backgrounds.
======
kopo
It's not simple to do.

First you need a whole lot of data to train the system which doesn't really
exist (unlike the case of speech to text or one language to another(text to
text)). You would need to set up a big data collection project.

Secondly there are a whole bunch of different things to track. Movement of
multiple fingers, +palm, +entire hand, +facial expressions. Current state of
the art would be tracking a single thing like a ball or a player for sports
and that still requires a number of cameras and a couple people sitting behind
the scenes fixing issues.

I think Microsoft's Kinect came close to achieving something akin to sign
language recognition at a very basic level. But from what I remember reading
they spent a whole lot of time and resources training their system just to get
there.

Maybe 3-4 years away I'd say.

------
laszlokorte
I am currently studying sign language in Germany (ie DGS - Deutsche Gebärden
Sprache, german sign language).

In the long term I can not think of anything why it should not be possible but
here a few difficulties that come to mind:

1\. Sign languages are in general not yet as well understood as spoken
languages (the grammar, the vocabulary). For example the University of Hamburg
is running the "DGS-Korpus" project[1] in order to create the first real
dictionary for DGS, currently there is none. i.e. even humans have not a full
explicit understanding of the language (DGS) and afaik it's similar for other
sign languages

2\. As video recordings are not older than 100 years and the community of
native speakers per language is much smaller than for spoken languages the
amount of recorded speech is much smaller

3\. Large parts of sign languages are very productive. That means that the
speaker/signer is not limited to a fixed vocabulary and a fixed grammar but
allowed to reenact a situation he is telling about ("constructed dialogue" and
"constructed action"[2]). Eg when retelling a conversation the storyteller can
shift her body to the left or the the right in order to embody multiple
people. Instead of saying "I drove the car. I had to turn left and then I saw
her on the right side" the signer can just move as he was sitting in the car,
holding the the steering wheel, moving it to the left and turning is head to
the right. I would compare that to making "phewww... ohhhhh, whoaaa" sounds in
order to tell how the plane you sit in took of. Additionally the signer is
allowed to allocate locations around her upper body to establish references to
previously mentions objects and refer back to them via pronouns.

4\. There is no agreed upon notation for writing down signed texts. Sure there
are notations for various hand forms and movements directions but especially
the productive part of language mentioned above is hard to formalize. E.g.
spacial locations that can be used for placing references are not predefined.
Some person still learning the language may only be able to use two positions
"left to his body" and "right to his body", but a very eloquent signer may be
able to establish even 10 references points in front of him and still enable
the audience to keep track of them. Such arbitrary reference points can not be
written down in a sensible notation.

5\. Various sign languages differ much more from each other than one might
initially think. Each sign language has its own grammar and vocabulary. For
german sign languages it's not even entirely clear what marks a sentence.

6\. Sign languages are not sequential but simultaneous languages ie it's
possible to express one thing with one hand and another thing with the other
hand in parallel

So some first steps might be:

* find a formal grammar for sign languages

* find a formal notation to write down arbitrary sentences

* collect a large corpus

* Train some AI to translate video data into your notation

* Train some AI to translate the notation from one sign language to another (and to spoken/written language as well)

* Train some AI to translate spoken/written language to your notation for some sign language

* Build an avatar that accepts that notation as input and translates it into movement

[1] [http://www.sign-lang.uni-hamburg.de](http://www.sign-lang.uni-hamburg.de)
[2]
[https://www.signteach.eu/index.php/podcastsall/item/construc...](https://www.signteach.eu/index.php/podcastsall/item/constructedaction-
dgs)

~~~
yorwba
> find a formal grammar for sign languages

Might not be necessary. Modern statistical machine translation techniques use
formal grammars as an auxiliary signal at most.

> find a formal notation to write down arbitrary sentences

Some sign languages might not be representable in SignWriting, but most should
be covered.
[https://en.wikipedia.org/wiki/SignWriting](https://en.wikipedia.org/wiki/SignWriting)

> collect a large corpus

Many (all?) European (and other) countries have laws requiring news
broadcasts, parliamentary debates etc. to be made available in sign language.
Similar to the EUROPARL corpus, that could be used as a starting point for
signed <-> spoken translation. Also similar to EUROPARL, colloquial language
would be underrepresented.

Of course there won't be any SignWriting transcriptions, so the corpus would
be essentially unlabeled.

> Build an avatar that accepts that notation as input and translates it into
> movement

I'd do that before/while attempting to create a large number of
transcriptions, both to ensure accuracy and to enable unsupervised pattern
mining in the later steps.

> Train some AI to translate video data into your notation

What I mean by unsupervised pattern mining is that the transcription AI would
be trained to output a transcription that makes the avatar ouput something
like the original video, as well as reconstructing the avatar's input from its
movements.

> Train some AI to translate the notation from one sign language to another
> (and to spoken/written language as well)

It'd probably be easiest to start with translating between the signed <->
spoken pairs in the corpus and then leverage that into signed <-> signed
translation.

~~~
laszlokorte
Yes you can do all that and as I said I do not think it would be impossible.
And some of what you describe is already done (eg [1]). I just wanted to point
out some difficulties.

[1]
[http://www.michaelkipp.de/slides/Kippetal11slides.pdf](http://www.michaelkipp.de/slides/Kippetal11slides.pdf)

