1 / 1

Asadullah Shah 1 , Aznan Zuhid Saidin 2 , Imad Fakhri Taha 1 , and Akram M. Zeki 2

12th Conference of the Pacific Association for Computational Linguistics - PACLING 2011 -. Frequencies Determination of Characters for Bahasa Melayu : Results of Preliminary Investigation. Asadullah Shah 1 , Aznan Zuhid Saidin 2 , Imad Fakhri Taha 1 , and Akram M. Zeki 2.

talisa
Download Presentation

Asadullah Shah 1 , Aznan Zuhid Saidin 2 , Imad Fakhri Taha 1 , and Akram M. Zeki 2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 12th Conference of the Pacific Association for Computational Linguistics - PACLING 2011 - Frequencies Determination of Characters for BahasaMelayu: Results of Preliminary Investigation Asadullah Shah1, Aznan Zuhid Saidin2, Imad Fakhri Taha1, and Akram M. Zeki2 1Department of Computer Science, 2Department of Information Systems, Kulliyyah of ICT, International Islamic University Malaysia. Email : asadullah@kict.iium.edu.my ABSTRACT Bahasa Melayu (Malay language) is a language spoken in Malaysia and many countries around it. It has rich literature and deep roots in culture. Bahasa Melayu language uses roman character set (i.e.A-Z) identical to English language. The written language uses the character set as building blocks to build word, sentences and phrases along with special punctuations and signs to create documents of interest. In this paper, results of preliminary investigation of Malay text documents are provided. For this purpose scanning of articles written upon various topics in Malay were carried out. Approximately 31 thousand characters from different articles are scanned. Preliminary observations indicate that on average, character “A” occurs 19%, character “N” occur 10%, character “E” occur “9%”and character “I” 8% in text. However, it is also observed from the data that, these are the characters from over all set with highest frequencies of occurances and it is expected that during further investigation they will remain as higher frequency occurring characters. Furthermore, the results indicate that for Bahasa Melayu characters appearance in text is very close in character frequencies of Bahasa Indonesia, but having different appearance of characters than English language. The investigation also indicate that these two languages, Bahasa Melayu and Bahasa Indonesia share close phonetic structure but not English, though all three use same character set. METHODOLOGY 24 articles of newspapers of various topics sizing from 1270 to 3665 characters per document are considered. Approximately 31 thousand characters are scanned. For each document the relative frequencies are observed, running averages are identified, and absolute frequencies of all characters from A-Z are calculated. RELATIVE FREQUENCIES Initial results of all 24 documents scanned indicate that four characters with the highest occurrence in Bahasa Malay are characters “A”, “N” and “E” and “I” are shown in Figures 1-4, respectively : Figure 4: for “I”, the relative probability varies between 0.062 to 0.105, i.e. the frequency varies between 6% to 10%. Figure 2: for “N”, the relative probability varies between 0.086 to 0.124, i.e. the frequency varies between 8% to 12%. Figure 3: for “E”, the relative probability varies between 0.059 to 0.111, i.e. the frequency varies between 5% to 11%. Figure 1: for “A”, the relative probability varies between 0.145 to 0.249, i.e. the frequency varies between 14% to 24%. ABSOLUTE FREQUENCIES RUNNING AVERAGE Running average is also called moving average or cumulative average, the formula as reported in [5] : where each x represents the total of a individual character in a document and i represent the total number of all documents. Figure 6: the running average frequency for “N” is stabilizing at the frequency of around 120. Figure 8: the running average frequency for “I” is stabilizing at the frequency of around 98. CONCLUSION The results show that the character with the highest frequency is “A”, followed by “N”, “E” and “I” respectively. It is predicted that this will remain true even if the number of documents are increased Characters “Q” and “X” have shown zero frequency in this study. Words in Bahasa Melayu having these characters are rare, mostly occurring in words originating from other languages. Issues mentioned here warrant for further studies. Figure 5: the running average frequency for character “A” is shown to be stabilizing at the frequency of around 230, which indicate that the frequency of “A” will somewhere stay around 230 in all 24 documents scanned. Figure 7: the running average frequency for “E” is stabilizing at the frequency of around 100. References : [1] S. Trost. (2011, March 14) Character frequency: Indonesian (Bahasa) [Online] Available: http://www.sttmedia.com/characterfrequency-indonesian. [2] Wikipedia. (2011, February 20) Letter frequency [Online] http://en.wikipedia.org/wiki/Letter_frequency. [3] WorldLingo. (2011, March 14) Letter frequency [Online] http://www.worldlingo.com/letter_frequency [4] A. Wang, “ First, second and Third order Entropies of Printed Malay”, Indian Journal of Statistics, ser. B, vol. 46, pt 3, pp 372-376, 1984. [5] Wikipedia. (2011, April 1) Moving Average [Online] http://en.wikipedia.org/wiki/Moving_average.

More Related