====================================================================== Language in India www.languageinindia.com ISSN 1930-2940 Vol. 18:10 October 2018 India’s Higher Education Authority UGC Approved List of Journals Serial Number 49042 ===================================================================== A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrase-based Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) Abstract The paper demonstrates the qualitative evaluation of the English to Urdu Machine Translation Systems, namely PBSMT and NMT hosted on Google’s Translate. This system is popularly known as Rosetta, formerly governed by Phrase-based approach and is presently governed by the neural module of source and target languages. In this study, a model corpus set of 100 English sentences has been applied out of 1k cross-domain data considering various types of verbs as input text to evaluate the output of the online systems in Urdu. In order to evaluate the output text in a qualitative manner, the Inter-translator Agreement (IA) of three human translators has been considered with their scores on a five-point scale. The scores are calculated by the Fleiss’ Kappa statistical measure with regard to comprehensibility and grammaticality on the basis of which error analysis and suggestions have been provided for improvement. The Kappa scores of PBSMT for comprehensibility and grammaticality are 0.24 and 0.22 respectively which is indicative of the fact that on both counts the scores are not up to the mark. Furthermore, the system has also been quantitatively evaluated on the basis of word error rate (21.11%) and sentence error rate (72.39%). On the contrary, NMT module has Kappa scores of 0.61 and 1 on comprehensibility and grammaticality respectively. So far as WER and SER are concerned, NMT has 32.58% and 28% respectively. In addition, all the erroneous entities have been analyzed through computational typology. The strategy for evaluation is to evaluate the Urdu output text based on the five-point scale with scores that range from 0-4 where 0 refers to incomprehensible or ungrammatical, 1 = little meaning or disfluent, 2 = neutrality, 3 = comprehensible or grammatical and 4 suggests flawless in both cases. Keywords: PBSMT, NMT, Google’s Translate, MT, Urdu, Indo-Aryan, NLP, Fleiss’ Kappa Overview As discussed in Castilho et al. (2018), since the advent of the Machine Translation (MT) or automated translation, new methods, approaches and techniques have really created high expectations among researchers. On one hand, the qualitative approaches have paved the way for a graded or incremental improvements in contrary to the significant improvements exhibited by the statistical approaches. Among the statistical techniques, Neural Machine Translation (NMT) has recently emerged as an innovative and robust technique as it has generated a lot of attention because of its high qualitative outputs in comparison to its counterparts. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 154 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems This study represents the qualitative evaluation of the English-Urdu Machine Translation systems namely Phrase-based Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) hosted on Google’s Translate. The system is also otherwise known as Rosetta earlier governed by the phrase-based model which is presently replaced by the neural model. The rationale for the consideration of the qualitative evaluation of the systems is that quantitative evaluation doesn’t prove to be adequate and sufficient in bringing out the reasons behind the error-prone outputs. Although every NLP MT platform is vying for adapting neural framework presently, it is not a panacea for all the issues in the domain. We have considered a representative corpus of 100 sentences out of 1k corpus ranging across various categories of verbs as input in English for the evaluation of Urdu output sentences. Machine Translation It is an automated translation process of text from source language (SL) to target language (TL). It is one of the sub-fields of Natural Language Processing (NLP) the sole objective of which is to enquire the application of the software for translating speech or text from one language to another one. MT Systems - A Review of Literature AnglaBharti is an English to Indian languages computer aided translation (CAT) system launched by Sinha et al. (1995) at Indian Institute of Technology, Kanpur in 1991. AnglaBharti-II was developed by Sinha in 2004 addressing shortcomings of the latter model and incorporating Generalized Example Base and Raw Example Base. Anubharti is a template-based Machine Translation of Hindi-English which applies Hybrid Example-based model which is an amalgamation of the strategies used in both the approaches of rulebased and example based for translation. Anusaaraka (1995) project was started by Prof. Rajeev Sangal and is presently the director at IIT BHU. It is a software which translates texts from English to Hindi languages. Anusaaraka is modelled upon Panini’s Ashtadhyayi which is based upon grammar rules and aims at mixing ancient Indian shastras and modern technologies. MaTra is a hybrid system trained on cross-domain corpus text and represents a pragmatic approach to language engineering. It is primarily utilized in the project on Cross Lingual Information Retrieval (CLIR) (Rao, 2001). AnglaHindi (Sinha, 2003) is an example-based English-Hindi version of AnglaBharti which can handle all types of sentences up to maximum 20 words each. It further integrates a rule and example-based approaches during the process of post-editing. Mantra, an English-Hindi MT trained specifically in the personal administration domain data, is developed by CDAC, Pune. The system applies Lexicalized Tree Adjoining Grammar (LTAG) which maps a lexical tree in SL to its counterpart lexical tree in TL. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 155 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems Shiva and Shakti are the Machine Translation systems jointly developed by Carnegie Mellon University, USA, IIIT, Hyderabad and IISc, Bangalore, India that translate texts from English to Hindi. Shakti MT system (Bharati et al., 2003) has been modelled and developed in such a way that they can produce MT systems for newly incorporated languages frequently and is hybrid in nature whereas Shiva is Example-Based in nature. Sampark MT system has been developed by 11 consortia member institutions in the project named ILMT project funded by the TDIL program of the DeiTY, Govt. of India. It has created NLP resources for 9 Indian language pairs resulting in Machine Translation for 18 languages. Anuvaadaksh system has been developed in the EILMT project, funded by the TDIL program of the DeiTY, Govt. of India which translates English text into 8 Indian languages. It has been conceptually designed and prepared by 13 institution consortium members and has integrated four MT technologies: Tree Adjoining Grammar, Statistical-base, EBMT and Analyze and Generate Rules. The first version of Microsoft’s MT system such as Bing Translator has been designed, developed and managed by Microsoft Research between the years 1999 and 2000. It was exploited to translate the whole gamut of Microsoft Knowledge Base into Spanish, French, German and Japanese. UCSG MAT is developed by the University of Hyderabad which is a machine-aided translation platform utilized in order for translating English texts as input into Kannada as output and also needs post-editing. Its primary purpose is to parse an English input sentence applying the UCSG parsing technology which was developed by Dr. K. Narayana Murthy and thereafter translates it into Kannada language applying the bilingual dictionary of English-Kannada, Morphological Generator of Kannada and the linguistic rules for translation. Universal Networking Language (UNL) is an international project of United Nations University in which IIT, Mumbai participates. It is an inter-lingua for semantic representation. Currently, this project is working in languages such as English, Hindi and Marathi where any of these languages is taken as SL and converted into UNL and then again de-converted from UNL to TL. Tamil Anusaaraka has been developed by K. B. Chandrasekhar Research Centre, Anna University, Chennai. Its primary aim was to develop a Human Aided Machine Translation System for the language pairs English-Tamil. It has three major components viz. morphological analyzer of English, mapping system unit and the Tamil language generator. MAT by Jadavpur University: Rule-based English-Hindi MAT is in Jadavpur University, Kolkata. It uses transfer-based approach and its purpose is to work for new sentences. Anuvaadak 5.0 system was developed for a general purpose of automatic translation from English-Hindi by Super Infosoft private limited, Delhi under the leadership and supervision of Mrs. A. R. Choudhury. For each specific domains like official, formal, agriculture, linguistics, technical and administrative, it contains inbuilt dictionaries. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 156 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems Anubaad Hybrid Machine Translation System was developed by Bandyopadhyay at Jadavpur University, Kolkata in 2004 for translating English news headlines into Bengali. The current version of the system works at the sentence level. Statistical MT was developed by IBM India Research Lab at New Delhi and its sole purpose was to translate texts between English and Indian languages. Oriya Machine Translation System (OMTrans) is developed by Utkal University, Vanivihar. The SL is English while TL is Oriya in this system. It serves the purpose of sense disambiguation using the N-gram model. Tamil-Hindi machine-aided translation system was developed by Prof. C. N. Krishnan at Anna University at KB Chandrashekhar (AU-KBC) Research Centre, Chennai. It is based on Anusaaraka Machine Translation System and applies a lexical level translation. Dr. K. Narayana Murhty has developed English-Kannada MAT system which is situated at Resource Centre for Indian Language Technology Solutions (RC-ILTS), University of Hyderabad. It is essentially based on a transfer-based approach which is applied to the documents related to government circulars. Hinglish machine translation system has been developed by Sinha and Thakur (2005) in 2004 which is based on pure Hindi to English forms. It has been executed after having incorporated an additional layer to the existing English to Hindi (AnglaBharti-II) and Hindi to English machine translation (AnuBharti-II) systems which was also developed by Sinha himself. English to (Hindi, Kannada, Tamil) and Kannada to Tamil language-pair example-based machine translation systems were developed by Balajapally et al. (2006). It encapsulates a bilingual dictionary which comprises of phonetic-dictionary, words-dictionary, phrases-dictionary and sentence dictionary. Punjabi to Hindi machine translation system was developed by Josan and Lehal at Punjabi University Patiala in 2007. It is based on a direct word-to-word translation mapping approach. Hindi to Punjabi machine translation system was conceptualized and developed by Goyal and Lehal (2010) at Punjabi University Patiala in 2009. It is also based on direct word-to-word mapping from SL to TL. Apni Urdu is an English-Urdu MT Platform which is an incorporation of the English-Urdu machine translated texts. As inputs, it applies some English texts which are readily available online and for outputs; it supports Urdu Unicode fonts. This platform is beneficial for simple constructions. The Apertium Machine Translation Platform was developed by Forcada et al. (2009) which provides a readymade framework for developing new platforms for any language pairs. Lavie et al. (2004) describe an MT system known as a trainable transfer-based Hindi-English MT platform which is designed to further the development of MT for less resourced languages. Behera et al. (2016a) have discussed about the divergence patterns between English to Bhojpuri language pairs where they have discussed about various syntactic and semantic divergences between English and Bhojpuri language pairs. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 157 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems Behera et al. (2016b) have proposed to transform the IMAGACT4ALL ontology into an MT platform where animated videos along with their sentential written expressions could be translated into other Indian languages either as visual representations or as written forms. Google Translate It was released by Google Inc. on 12 Jan, 2010. Its revised version came in 2011. This platform is providing services for many languages around the globe. Among Indian languages, it supports Hindi, Urdu, Bengali, Gujarati, Punjabi, Telugu, Tamil, Kannada, Marathi, Malayalam and Nepali (Muzaffar and Behera, 2014). Phrase-based Statistical Machine Translation It is modeled upon the phrase-based language translation model which translates the SL phrase into the TL phrase. The PBSMT used is Moses (Koehn, 2009), MGIZA (Gao and Vogel, 2008) is applied for training word alignments, and KenLM (Heafield, 2011) is applied to language model training and scoring. It is a linear combination of different features such as phrase and word penalty, 5-gram language model and phrase translation probabilities with some modification techniques of Kneser-Ney smoothing (Kneser and Ney, 1995; Chen and Goodman, 1998) and the below mentioned advanced features such as a 5-gram operation sequence model (Durrani et al., 2013); a hierarchical lexicalized reordering model (Galley and Manning, 2008); sparse features which indicate phrase pair frequency, length of phrase, and sparse lexical features. For English-Russian pair of languages, it employs a transliteration mapping model for unknown sort of words (Durrani et al., 2014). Feature weights optimization is applied in order to increase the level of BLEU score with the batch MIRA (Cherry and Foster, 2012) on a within-domain tuning set that has been extracted (and held out) from the in-domain training data. Muzaffar and Behera (2014) have provided a detailed description on the errors pertaining to verb markers in Urdu while dealing with the translation platform of Google and Bing. Muzaffar et al. (2016a) have proposed a parser based on Pāniniān framework for successfully analyzing errors related to case markers in English-Urdu Machine Translation in general. Muzaffar et al. (2016b) have provided a detailed description on the divergent patterns between English and Urdu after observing the outputs collected from Google and Bing MT platforms. Muzaffar and Behera (2016c) have dealt with the concepts of equivalence, gain and loss in Machine Translation while experimenting on Google and Bing. Gupta and others (2013) have conducted both subjective and objective evaluations of English to Urdu Machine Translation. Neural Machine Translation It focuses on semantics of SL and TL and thus semantically makes efficient translation than PBSMT. It involves building a single neural network that maps SL and TL aligned bilingual texts and is designed and trained so as to “maximize the probability of a correct translation” (Bahdanau et al., 2014), when given input text to translate without external linguistic information. This interest is shared by many in the language service industry, where there is a need for improved MT quality and better quality estimation to “help reduce the frustrating aspects of post-editing” (Etchegoyhen et al., 2014). NMT results in the latest shared tasks have quickly matched or surpassed those of PBSMT systems, even after so many years of development of PBSMT systems (Sennrich et al., 2017; Bojar et al., 2016). As according to the reported recent studies on NMT, it can be vehemently affirmed that one can observe a significant increase in quality if one considers the comparison of NMT with PBSMT applying either automatic (Bahdanau et al., 2014; Jean et al., 2015), or human evaluations (Bentivogli et al., 2016; Wu et al., 2016). Although the initial NMT experiments have exhibited a significant increase in results, human evaluations on NMT output have not been conducted on a large-scale basis. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 158 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems Urdu Language Urdu is an Indo-Aryan language (Muzaffar et al, 2015) and is a member of the New-IndoAryan which is a subgroup of Indo-European family of languages. It is spoken in most of the areas of Indian sub-continent (Muzaffar & Behera, 2014). According to the census 2011, there are approximately fifty million speakers of Urdu in India. Methodology This section has been divided into corpus collection method and data analysis method. Method of Corpus Collection We have collected 1000 corpus of English sentences selectively considering different types of verbs. Out of them, 100 representative sentences have been taken as input to evaluate the output data in Urdu. Method of Data Analysis Data has been analysed by considering the Inter-translator Agreement (IA) of three human translators with their scores on a five-point scale which range from 0-4 where 0 stands for incomprehensible or ungrammatical, 1 means little meaning or disfluent, 2 refers to neutrality, 3 stands for comprehensible or grammatical and 4 suggests flawless. The scores are calculated by the Fleiss’ Kappa statistical measure with regard to comprehensibility and grammaticality on the basis of which error analysis and suggestions have been provided for improvement. Evaluation Evaluation is considered to be one of the stepping stones for measuring the efficiency of an NLP application (Mitkov, 2003). It can be of two broad categories: human and automatic or statistical based on two approaches to research i.e. qualitative and quantitative. Qualitative vs Quantitative Evaluation In qualitative evaluation, judgments of different translators have been considered for measuring the output texts of the MT systems. Contrarily, the quantitative evaluation conducts a statistical measurement of the performance. Therefore, it is indispensable that we need to perform a qualitative evaluation of the systems so as to figure out their performance and various bottlenecks constricting the efficiency (Behera et al., 2016). The Role of Qualitative Evaluation Qualitative evaluation leaves us some room, for it evaluates any system with regard to reliability and adequacy or comprehensibility and acceptability, and so on. It gives us much background to the underlying issues and challenges as to why a given system under performs. Thus, we have undertaken a detailed qualitative evaluation of Google Translate where we have taken English as the SL and Urdu as the TL. Evaluation has been conducted at three levels: percentage agreement, Fleiss’ Kappa Agreement, WER and SER. Fleiss’ Kappa “Kappa” is a statistical measure applied to test the inter-rater reliability judgements between two or more raters qualitatively. There are two types of Kappa: e.g. Cohen and Fleiss. The former is ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 159 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems applied to the evaluation of the output for the agreement between two persons. On the other hand, the latter is applied to the agreement on the evaluation among multiple raters. In 1971, Fleiss extended Cohen’s Kappa for measuring IA reliability of more than two raters. Fleiss’ Kappa is defined as the output of the agreement above chance divided by the degree of agreement actually achieved. It takes values between 0-1 where 1 signifies complete agreement. The scores of three native translators according to Fleiss ’Kappa have been taken into account. Categories Qualitative/Human Comprehensibility Grammaticality pa pe K pa pe K Urdu (PBMT) 0.450 0.270 0.246 0.450 0.286 0.229 Urdu (NMT) 0.7 0.22 0.61 1 0.37 1 Table 1. Qualitative Evaluation of Google’s MT Platform on Kappa Statistics On one hand, the Kappa score of PBSMT for comprehensibility is 0.24 and grammaticality is 0.22 which are not up to the mark. On the other hand, NMT has Kappa scores of 0.61 for comprehensibility and 1 for grammaticality. The Kappa scores of PBSMT and NMT for comprehensibility are 0.24 and 0.61 respectively. A comparatively higher score of NMT depicts that it performs better than PBSMT in terms of comprehensibility. So far as grammaticality is concerned, PBSMT has 0.22% and NMT has 1%. The score of NMT suggests to the fact that it has almost the perfect agreement among raters and thereby there are no erroneous patterns observed. Quantitative/Statistical (PBMT) 1 WER SER2 Quantitative/Statistical (NMT) WER SER 21.11% 32.58% 72.39% 28.00% Table 2. Quantitative Evaluation of Google’s MT Platform on WER & SER In this section, the higher number of scores is proportionate to the higher amount of erroneous linguistic patterns at the corresponding level. PBSMT and NMT have further been quantitatively evaluated on the basis of word error rates: 21.11% and 32.58% respectively. WER being higher for NMT implies that at the word level, the PBSMT outperforms its counterpart. So far as sentence error rates are concerned, PBSMT has 72.39% whereas NMT has 28.00%. The SER score being higher for the PBSMT is indicative of the fact that NMT outperforms its counterpart at the sentence level. PBSMT NMT Computational Errors 2.5% 1.8% Table 3. Distribution of Computational Errors 1 2 Word Error Rate Sentence Error Rate ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 160 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems The scores of Computational Errors on PBSMT and NMT are 2.5% and 1.8% respectively. A comparatively lower score of NMT refers to the fact that it is less-prone to errors and thus performs better than the PBSMT. Analysis of Computational Errors Tokenization It is one of the computational processes of text segmentation which classifies different characters from the other preceding and following ones. In the following example, the apostrophe is not usually translated in most of the instances. The following representative example exhibits that although the genitive form is translated properly, the plural oblique form (laDakI-yoM) has not been translated from the English counterpart. English: Girls’ college (apostrophe + plural) Itrans: laDakI ke mezabAna Named Entities (‘institute’ missing) Named entities are the proper nouns of any language. In the below-instantiated example, the part ‘institute’ is missing when translated from English to Urdu. Firstly, the Urdu output suggests that it is merely a transliteration output and not a translation of the original SL input text. Furthermore, the word ‘institute’ is not translated at all. English: All India Institute of Medical Sciences Itrans: Al InDiA medikAl sAIns Morphological These errors pertain to both the noun and verb morphology of Urdu getting wrongly translated. In the below example, ‘left over’ English adjectival phrase is getting erroneously translated into Urdu as a verbal phrase. English: left over pieces of food Itrans: khAne ke TukaDe TukaDoM para ChoDa diyA Chunking Chunking is the computational process of grouping the local words, viz. nominal categories and verbal categories to be grouped under their respective single broad category. Here, in the belowstated example, ‘call off’ as a verbal phrase has been inverted and hence causing problems during translations. English: I am calling the meeting off. Itrans: maiM mulAqAta kara rahA huM. Parsing (covert you) It is the computational process of having syntactic relations between and among different parts of the sentences. In the following example, the covert you in both the sentences has not been translated appropriately. English: Silence please!!! Itrans: barAye karam qhAmoshI Multi-words (idioms) ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 161 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems It is one of the computational categories, the meaning of which is different from their constituent parts. There are many sub-categories such as idioms and phrases, reduplications, echoword formations. In the following instance, the idiomatic expression has been literally translated. English: Pull up the socks Itrans: jarAbeM khichoM Conclusion One of the limitations of the current research is that we have taken into consideration only the computational errors for the purpose of analysis. Another most important limitation is that we have applied a limited amount of data of general domain and fed the Google MT system for this study. Depending upon the specificity and nature of the domain of the data, there will definitely be positive or negative impact on the quality of the output. In this paper, we have presented a qualitative evaluation of Google’s PBSMT and NMT between English and Urdu. We have applied the Fleiss’ Kappa method to measure agreement among multiple raters-cum-translators. We have further demonstrated analysis of errors computationally. We have compared Google’s PBSMT and NMT platforms and have observed that NMT performs well for this pair of languages. We would further like to replicate and extend this study to other Indian languages. We would like to reiterate a point that certainly NMT outperforms the PBSMT on the yardstick of the qualitative nature of the TL output. But it is not a panacea for all the issues and challenges pertaining to MT. For machines to perform at par with the humans or outperform them, let’s say, Machine Learning and NLP have to go a long way. Acknowledgements We are indebted to Google’s Translate developed by Google Inc. for making it readily available on the web. ================================================================= References Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Behera, P., Singh, R. & Jha, G. N. (2016). Evaluation of Anuvadaksh (EILMT) English-Odia Machine-assisted Translation Tool. In Proceedings of WILDRE-3 (LREC-2016), Portoroz, Slovenia, pp. 110-117. Behera, P., Maurya, N., Pandey, V. & Banerjee, E. (2016a). Dealing with Linguistic Divergences in English-Bhojpuri Machine Translation. In Proceedings of WSSANLP-6, COLING-2016, Osaka, Japan. ____________, Muzaffar, S., & Jha, G. (2016b). The IMAGACT4ALL ontology of animated images: implications for theoretical and machine translation of action verbs from English-Indian languages. In Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016) (pp. 64-73). Bentivogli, L., Bisazza, A., Cettolo, M., & Federico, M. (2016). Neural versus phrase-based machine translation quality: a case study. arXiv preprint arXiv:1608.04631. Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., ... & Negri, M. (2016). Findings of the 2016 Conference on Machine Translation. In ACL 2016 First Conference on Machine Translation (WMT16) (pp. 131-198). The Association for Computational Linguistics. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 162 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems Castilho, S., Moorkens, J., Gaspari, F., Sennrich, R., Way, A., & Georgakopoulou, P. (2018). Evaluating MT for massive open online courses. Machine Translation, 1-24. Durrani, N., Haddow, B., Heafield, K., & Koehn, P. (2013). Edinburgh’s machine translation systems for European language pairs. In Proceedings of the Eighth Workshop on Statistical Machine Translation (pp. 114-121). Durrani, N., Koehn, P., Schmid, H., & Fraser, A. (2014). Investigating the usefulness of generalized word representations in SMT. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers (pp. 421-432). Dwivedi, S. K., & Sukhadeve, P. P. (2010). Machine translation system in Indian perspectives. Journal of computer science, 6(10), 1111. Etchegoyhen, T., Bywood, L., Fishel, M., Georgakopoulou, P., Jiang, J., Van Loenhout, G., ... & Volk, M. (2014). Machine Translation for Subtitling: A Large-Scale Evaluation. In LREC (pp. 4653). Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): 378. Forcada, M. L., Ginestí-Rosell, M., Nordfalk, J., O’Regan, J., Ortiz-Rojas, S., Pérez-Ortiz, J. A., ... & Tyers, F. M. (2011). Apertium: a free/open-source platform for rule-based machine translation. Machine translation, 25(2), 127-144. Garje, G. V., & Kharate, G. K. (2013). Survey of Machine Translation Systems in India. International Journal on Natural Language Computing (IJNLC), 2(4), 47-67. Gao, Q. and Vogel, S. (2008). Parallel implementations of word alignment tool. In Software Engineering, Testing, and Quality Assurance for Natural Language Processing. Association for Computational Linguistics, 49–57. Goyal, V., & Lehal, G. S. (2010). Web based Hindi to Punjabi machine translation system. Journal of Emerging Technologies in Web Intelligence, 2(2), 148-151 Gupta, V., Joshi, N. & Mathur, I. (2013, August). Subjective and objective evaluation of English to Urdu Machine translation. In Advances in Computing, Communications and Informatics (ICACCI), pp. 1520-1525. IEEE. Heafield, K. (2011, July). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation (pp. 187-197). Association for Computational Linguistics. Jean, S., Firat, O., Cho, K., Memisevic, R., & Bengio, Y. (2015). Montreal neural machine translation systems for WMT’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation (pp. 134-140). Koehn, K. (2009). Statistical Machine Translation. Cambridge University Press. Mitkov, R. (2003). The Oxford Handbook of Computational Linguistics. Oxford University Press, New York. Muzaffar, S. & Behera, P. (2014). An Error Analysis of the Urdu Verb Markers: A Comparative Study on Google and Bing Machine Translation Platforms. Aligarh Journal of Linguistics, 4 (1-2), pp. 199-208. ____________, Behera, P., Jha, G. N., Hellan, L. & Beermann, D. (2015). The TypeCraft Natural Language Database: Annotating and Incorporating Urdu. Indian Journal of Science and Technology, 8(27), October, Chennai, India. ____________, Behera, P. & Jha, G. N. (2016a). A Pāniniān Framework for Analyzing Case Marker Errors in English-Urdu Machine Translation. Procedia Computer Science (Elsevier), 96, pp. 502-510. ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 163 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems ____________, Behera, P. & Jha, G. N. (2016b). Classification and Resolution of Linguistic Divergences in English-Urdu Machine Translation. In Proceedings of WILDRE-3 (LREC-2016), (ISBN: 978-2-9517408-8-4), Portoroz, Slovenia, pp 43-49. ____________. & Behera, P. (2016c). The Concepts of Equivalence, Gain and Loss (Divergence) in English-Urdu Web-Based Machine Translation Platforms. In Proceedings of International Conference of South Asian Languages (ICOSAL-12), CELMTS, University of Hyderabad. Sennrich, R., Firat, O., Cho, K., Birch, A., Haddow, B., Hitschler, J., ... & Nădejde, M. (2017). Nematus: a toolkit for neural machine translation. arXiv preprint arXiv:1703.04357. Sinha, R. M. K., & Jain, A. (2003). AnglaHindi: an English to Hindi machine-aided translation system. MT Summit IX, New Orleans, USA, 494-497. Sinha, R. M. K. (2004, November). An engineering perspective of machine translation: anglabharti-II and anubharti-II architectures. In Proceedings of International Symposium on Machine Translation, NLP and Translation Support System (iSTRANS-2004) (pp. 10-17). Sinha, R. M. K., & Thakur, A. (2005). Machine translation of bi-lingual hindi-english (hinglish) text. 10th Machine Translation summit (MT Summit X), Phuket, Thailand, 149-156. Sinha, R. M. K., Sivaraman, K., Agrawal, A., Jain, R., Srivastava, R., & Jain, A. (1995, October). ANGLABHARTI: a multilingual machine aided translation project on translation from English to Indian languages. In Systems, Man and Cybernetics, 1995. Intelligent Systems for the 21st Century, IEEE International Conference on (Vol. 2, pp. 1609-1614). IEEE. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., ... & Klingner, J. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ================================================================== Pitambar Behera, M.A., Sharmin Muzaffar, M. B.Ed., M.Phil., Ph.D. (Linguistics) Centre for Linguistics A., Ph.D. (Linguistics) School of Language, Literature and Culture Department of Linguistics, Faculty of Arts Studies Aligarh Muslim University Jawaharlal Nehru University Aligarh, Uttar Pradesh-202002, India New Delhi-110067, India [email protected] ==================================================================== Language in India www.languageinindia.com ISSN 1930-2940 18:10 October 2018 Sharmin Muzaffar, M. A., Ph.D. (Linguistics) Pitambar Behera, M.A., B.Ed., M.Phil., Ph.D. (Linguistics) 164 A Qualitative Evaluation of Google’s Translate: A Comparative Analysis of English-Urdu Phrasebased Statistical Machine Translation (PBSMT) and Neural Machine Translation (NMT) Systems