Checking what online machine translation service is best for you
Some claim that learning foreign languages is a waste of time, that translators are soon to disappear from the professional market and that technology can get you from language A to language B in no time, for free and without any trouble. Myth or reality? The debate is open; however it is true that technology can be a helpful tool and a myriad of online translation software – also known as machine translation system or “MT” – can be found on the web. So many that it may be challenging to find the right MT for the right text. This is why we have a run a cross-comparative analysis between four MT – Systran’s Babel Fish, Google Translate, PROMT and Microsoft’s Bing Translator and using five languages.
The set-up: Turn on the machines!
Machine translation technology is a complex science. There are many types of MT, among others statistical, hybrid, rule-based or sentence-based. We decided to focus on the user’s perspective and test the four MT efficiency, leaving the technical part aside. If you wish to learn more about the different types of MT and how they work, please check Hutchins and Somers.
In order to carry out this experiment, we gathered a corpus of 500 sentences, submitted to the aforementioned four MT. Ten language combinations were put on trial – 50 sentences per combination. These are English-French, English-German, English-Spanish, English-Italian, English-Portuguese and reversed. Each sample of 50 sentences was translated via each of the four MT and their respective results evaluated on a scale from 0 to 3. 0 for untranslated or not understandable results, 1 when the meaning had to be guessed, 2 when the gist was correct with grammatical mistakes and 3 for translation that would almost compete with the work of a professional translator. Each MT could gather up to 120 points per language combination evaluated.
Because we wanted the corpus of sentences to be as well-balanced as possible, we selected 5 sentences of 10 different areas in each batch. These are advertisement, business, financial, gastronomic, legal, literature, medical, religious, slang and Tweet. Each of those features their own difficulties when it comes to translation. This is why we designed a breakdown of their respective scores for each language combination in order to find out which domains are well-handled by MT and which ones are not.
Overall results: Who wins the battle?
Here are the results that we obtained:
Unit: points (out of a maximum attainable of 120)
The results show interesting trends. Overall, Google Translate seems to be providing a better translation than the other MT, followed by Bing Translator and Systran and PROMT at the end. The only instance where Bing scored highest is for Spanish-English. Some language combinations such as French-English, Italian-English or Portuguese-English managed to spawn relatively good results; Google Translate gathered over 80 points out of 120 in these combinations – close to 75% of perfect translation. However, Spanish MT results were across all four of them very weak. Likewise, German turns out to be more challenging to translate, as source text and as target text alike. Winner: Human translation. Runner-up: Google Translate.
The chart below shows how well each MT performed across all languages and how many point they gathered out of a total of 1,200. Because some language combinations, as well as some areas of translation, are so challenging the overall results are very average: only a third of the text produced is correct for PROMT and Sytran, about half of it for Bing and Google. There is still significant progress to be done before MT can be used with perfect reliability and accuracy.
Unit: points (out of a maximum attainable of 1,200)
Digging deeper: Results across subjects areas
From the graphs below, we can identify some trends. On the “winning” side, medical translation (probably due to the straight-forward descriptive texts) is what can be best handled by MTs, being in the top 3 in 7 out of 10 language combinations. Advertisements are generally well translated as well – 6 times in the top 3, once in the bottom 3. This is generally because ads use simple sentences easy to remember. Finally, recipes (gastronomic) appear 5 times in the top 3 twice in the bottom 3. This area did especially well when English is the source language as recipes are written in the imperative mode in English, an easy mode for MT to handle.
Looking at the bad players of the experiment, literary translation is, without much surprise, the worst case scenario for MT: elaborated syntax, rare words and unusual figures of speech, among others. As a result, literature made to the bottom 3 in 9 of the combinations. 7 times in the bottom free – both slang and Tweets. Slang uses metaphorical speech, which is usually translated literally by the MT, resulting in a nonsensical segment. Tweets contain exclusive elements the other sentences do not have, such as colloquial abbreviations, URL or symbols (#, @). These sometimes induced MT to make mistakes, in addition to the often bad quality of language used on Twitter: abbreviations, slang words, etc. See the detailed results on the graphs below.
Regardless of what you need to translate, do not expect perfection from MT. They can and do produce intelligible results in most cases, given that the source text is basic, well-written and not too ambiguous, if at all. Out of the four MT tested, Google Translate turned out to be the best one to use in most occurrences. Texts that are generally speaking factual (medical, gastronomical) are MT-friendlier than creative writing (literature, Tweets). Plain texts and simple sentences are consequently easier for MT as they avoid multiple possible translations – this is known as controlled language. Finally, we would like to advise you against relying solely on MT for important subjects or published texts – that could bring you serious embarrassments!
