The Investigation of Convolution Layer Structure on BERT-C-LSTM for Topic Classification of Indonesian News Headlines
DOI:
https://doi.org/10.17509/seict.v4i2.63742Keywords:
BERT, Convolution Layer, IndoBERT, LSTM, News, Word EmbeddingAbstract
An efficient and accurate method for classifying news articles based on their topics is essential for various applications, such as personalized news recommendation systems and market research. Manual classification methods are tedious, prompting the use of deep learning techniques in this study to automate the process. The developed model, BERT-C-LSTM, combines BERT, the convolutional layer from CNN, and LSTM, leveraging their individual strengths. BERT excels at transforming text into context-dependent vector representations, The design of the classification model employs a blend of convolutional layers and LSTM, referred to as C-LSTM. The convolutional layer possesses the capability to extract salient elements, including keywords and phrases, from input data. On the other hand, the Long Short-Term Memory (LSTM) model exhibits the ability to comprehend the temporal context present in sequential data. This study aims to investigate the influence of the convolutional layer structure in BERT-C-LSTM on the classification of Indonesian news headline categorized into eight topics. The results indicate that there are no significant differences in accuracy between BERT-C-LSTM model architectures with a single convolutional layer and multiple parallel convolutional layers and the models using various filter sizes. Furthermore, the BERT-C-LSTM model achieves an accuracy that is not much different from the BERT-LSTM and BERT-CNN models, with accuracies reaching 92.6%, 92.1%, and 92.7%, respectively.References
Chen, Y., Perozzi, B., Al-Rfou, R., & Skiena, S. (2013). The Expressive Power of Word Embeddings. 28.
Chowdhury, P., Eumi, E. M., Sarkar, O., & Ahamed, M. F. (2022). Bangla News Classification Using GloVe Vectorization, LSTM, and CNN. Lecture Notes on Data Engineering and Communications Technologies, 95(December 2021), 723–731. https://doi.org/10.1007/978-981-16-6636-0_54
Dong, J., He, F., Guo, Y., & Zhang, H. (2020). A commodity review sentiment analysis based on BERT-CNN model. 2020 5th International Conference on Computer and Communication Systems, ICCCS 2020, 143–147. https://doi.org/10.1109/ICCCS49078.2020.9118434
Ezen-Can, A. (2020). A Comparison of LSTM and BERT for Small Corpus. 1–12.
Fauzi, M. A. (2018). Automatic Complaint Classification System Using Classifier Ensembles. Telfor Journal, 10(2), 123–128. https://doi.org/10.5937/telfor1802123A
González-Carvajal, S., & Garrido-Merchán, E. C. (2020). Comparing BERT against traditional machine learning text classification. Ml.
Ingole, P., Bhoir, S., & Vidhate, A. V. (2018). Hybrid Model for Text Classification. Proceedings of the 2nd International Conference on Electronics, Communication and Aerospace Technology, ICECA 2018, Iaeac, 450–458. https://doi.org/10.1109/ICECA.2018.8474920
James Mutinda, Mwangi, W., & Okeyo, G. (2023). Sentiment Analysis of Text Reviews Using Lexicon-Enhanced Bert Embedding (LeBERT) Model with Convolutional Neural Network. Appl. Sci, 13(1445). https://doi.org/10.3390/app13031445
Kandhro, I. A., Jumani, S. Z., Kumar, K., Hafeez, A., & Ali, F. (2020). Roman Urdu Headline News Text Classification Using RNN, LSTM and CNN. Advances in Data Science and Adaptive Analysis, 12(02), 2050008. https://doi.org/10.1142/s2424922x20500084
Kaur, K., & Kaur, P. (2023). BERT-CNN: Improving BERT for Requirements Classification using CNN. Procedia Computer Science, 218(2022), 2604–2611. https://doi.org/10.1016/j.procs.2023.01.234
Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP. 757–770. https://doi.org/10.18653/V1/2020.COLING-MAIN.66
Li, W., Gao, S., Zhou, H., Huang, Z., Zhang, K., & Li, W. (2019). The automatic text classification method based on bert and feature union. Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS, 2019-Decem, 774–777. https://doi.org/10.1109/ICPADS47876.2019.00114
Maslennikova, E. (2019). ELMo Word Representation For News Protection.
Murfi, H., Gowandi, T., Ardaneswari, G., & Nur-, S. (2022). BERT-Based Combination of Convolutional and Recurrent Neural Network for Indonesian Sentiment Analysis. 1–15.
Pogiatzis, A. (2019). NLP: Contextualized word embeddings from BERT.
Safaya, A., Abdullatif, M., & Yuret, D. (2020). KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media. 14th International Workshops on Semantic Evaluation, SemEval 2020 - Co-Located 28th International Conference on Computational Linguistics, COLING 2020, Proceedings, 2054–2059. https://doi.org/10.18653/v1/2020.semeval-1.271
Wang, J., Wang, Z., Zhang, D., & Yan, J. (2017). Combining knowledge with deep convolutional neural networks for short text classification. IJCAI International Joint Conference on Artificial Intelligence, 0, 2915–2921. https://doi.org/10.24963/ijcai.2017/406
Widhiyasana, Y., Semiawan, T., Gibran, I., Mudzakir, A., & Noor, M. R. (2021). Penerapan Convolutional Long Short-Term Memory untuk Klasifikasi Teks Berita Bahasa Indonesia (Convolutional Long Short-Term Memory Implementation for Indonesian News Classification). Jurnal Nasional Teknik Elektro Dan Teknologi Informasi |, 10(4), 354–361.
Zhang, Z., Zhao, H., & Wang, R. (2020). Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond.
Zhou, C., Sun, C., Liu, Z., & Lau, F. C. M. (2015). A C-LSTM Neural Network for Text Classification. October 2017.
Zhou, H. (2022). Research of Text Classification Based on TF-IDF and CNN-LSTM. Journal of Physics: Conference Series, 2171(1). https://doi.org/10.1088/1742-6596/2171/1/012021
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Journal of Software Engineering, Information and Communication Technology (SEICT)

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
