I just returned from a trip down to Mexico for my best friend’s wedding where, despite many shocking sights and sounds, the most astonishing thing I experienced was the Google Translate word lens in action. As a group of us sat down at a local eatery and began to browse the menu written in Spanish, a friend pulled out her phone and hovered over the words with the Google Translate live camera. I watched as the words appeared right before my eyes in freakishly accurate English. We were all dumbfounded and began to come up with theories for how this could possibly work so quickly and so well.
After tossing around a couple of outlandish guesses, someone blurted out the word ‘algorithm.’ Yes, the ubiquitous buzz word I can’t seem to escape. Then, my piqued curiosity lead me to further explore how the app works. While researching the ‘word lens,’ I found that the Japanese to English translation was just made available, and the Spanish to English translation had been around for some time already. I dug a bit deeper to understand how this technology operates and this is what I found.
Google Neural Machine Translation
Perhaps the most exciting aspect of this Google Translate tool was the instantaneous, largely accurate English translation of text from a foreign language. I found that the translation error has been dramatically reduced because of the more recent introduction of the Google Neural Machine Translation (GNMT). This machine learning algorithm analyzes full sentences and words in context rather than individual words or phrases on their own which is what Phrase Based Machine Translation (PBMT) does. GNMT has reduced translation errors by more than 55-85% on several major language pairs. From what I can tell, GNMT has totally transformed Google Translate into a dependable, mobile translation tool. But the major functionality game changers in creating the app are artificial intelligence and virtual reality.
stable|kernel website through word lens
Artificial Intelligence
As a matter of fact, all of the translations come from computers that use a process called ‘statistical machine translation.’ This means computers generate translations based on patterns found in large amounts of text. Google programmed computers to analyze millions of documents that had been translated by human translators – from books, organizations and websites around the world. These computers scan the text for patterns, and once the computer identifies a pattern, it can use that same pattern to translate similar text in the future. The artificial intelligence has a growing set of translated texts to scan for patterns, therefore increasing its knowledge base and capacity for translations of all languages with sufficient translation documentation. This is how the text is immediately translated when the camera hovers over the foreign language. The artificial intelligence summons the infinite library of patterns and translates on the spot.
Additionally, a system called ‘Zero Shot Translation’, which is a translation between language pairs never seen explicitly by the system, has proven even more intelligent than the original artificial intelligence. Enabling this technology allows the system to translate between a language pair it’s never seen before! It does this by essentially creating its own representational language or ‘interlingua.’ This language is a way for the system to produce and make sense of newly interpreted information. For example, Zero Shot Translation can translate between Korean and Japanese languages even though it’s never been taught to do so.
Related: The Technology Behind Pokemon Go: Why Augmented Reality is the Future
Virtual Reality
The other equally impressive feature of the word lens function in the app is the almost magical wording that appears before your eyes through the camera in place of the original text. This is where virtual reality comes in. As the words are translated in a blink of an eye, virtual reality superimposes translated text directly on the foreign text using the camera phone. Even more miraculous? You don’t need a network connection or WiFi.
As an insatiable traveler who can be intimidated by language barriers, I am so pleased to have such a pragmatic tool at my disposal. Thanks to this beautifully-built piece of software, I am excited for my next trip abroad. Whether Korea, Japan, Brazil or maybe Mexico again, I say…
