Experts at the University of Amsterdam are developing a computer programme that can translate into sign language, using the power of artificial intelligence.

The project, part of SignLab Amsterdam, aims in the first instance to support parents of deaf children who are learning basic signing or for situations like announcements at airports and railway stations.

‘Over the next five years I’m pretty confident we can provide something that will be useful and hopefully make a big difference,’ said Dr Floris Roelofsen, associate professor at the university’s institute for logic, language and computation and founder of the lab.

‘Six years ago, my daughter was born deaf and that’s how I became interested in sign language and deaf issues and culture. Most parents of deaf children are hearing, and it’s very difficult to learn sign language fast and well enough to start communicating with your child. It’s a big issue: if you don’t have language, there’s a delay in many other domains [like] cognitive and social [development]. I think this is the root of many problems in the lives of deaf children and adults.’

Prototype

Drawing on his own background in maths, artificial intelligence, philosophy and linguistics, Roelofsen has won the funding for a five-year project to develop an avatar that can translate simple sentences and instructions into animations.

The project, at prototype stage has already been trialled this year as a machine translation tool to improve the communication between healthcare professionals and deaf patients suffering from the coronavirus. The avatar, who does not yet have a name, can produce around 8,000 sentences such as ‘where are you feeling pain?’ or ‘the MRI is not entirely good.’

Roelofsen believes that the tool could also be used to help parents of deaf children learn sign language and sentence construction, because Dutch sign language creates a kind of ‘picture’ of events, rather than conjugating verbs.

‘In English you’d say “I go, I went or I will go,”’ he said. ‘[In sign language] it’s just one sign, go, but separately you indicate the time: “Yesterday I store go”. You first create the scene in space, then you put the actors and the objects, then you indicate what the action was or the relation between the objects. You do scene building.’

Train

One potential use, which has already been trialled at a separate experimental project at the University of Zurich, is to use an avatar to translate a set of potential train or airport announcements into signs.

‘This is something many deaf people recognise,’ said Roelofsen. ‘They are on a platform and all of a sudden everybody goes somewhere else and they don’t know why because they didn’t hear it. At the train station you can’t record every possible announcement [in a sign language video] but you want a combinatory system that can combine different signs together spontaneously. It still amounts to millions of possible announcements but the vocabulary and structure of the announcement is very restricted.

‘Train stations and airports would have a huge impact, and also websites. Many websites have some form of information that could be translated, and you cannot create videos for the whole web.’

However, he said, the artificial intelligence certainly does not mean that figures like the popular sign language interpreter Irma Sluis, present on many coronavirus government press conferences, would be replaced.

‘Most people think it would be applied in interactive dialogue but that’s way beyond the potential of current technology,’ he said. ‘It’s something we might dream of but it’s 50 years ahead. It’s not the ambition to replace Irma or any other human interpreter.’