IJIEEB Vol. 17, No. 2, 8 Apr. 2025
Cover page and Table of Contents: PDF (size: 774KB)
PDF (774KB), PP.51-67
Views: 0 Downloads: 0
IPO, BERT, Multilingual Tweet Processing, Sentiment Analysis, Financial Social Media, Natural Language Processing (NLP)
There is a growing interest in multilingual tweet analysis through advanced deep learning techniques. Identifying the sentiments of Twitter (currently known as X) users during the IPO (Initial Public Offering) is an important application area in the financial domain. The number of research works in this domain is less. In this paper, we introduced a multilingual dataset entitled as LIC IPO dataset. This work also offers a modified majority voting-based ensemble technique in addition to our proposed dataset. This test-time ensembling technique is driven by fine-tuning of state-of-the-art transformer-based pretrained language models used in multilingual natural language processing (NLP) research. Our technique has been employed to perform sentiment analysis over LIC IPO dataset. Performance evaluation of our technique along with five transformer-based multilingual NLP models over this dataset has been reported in this paper. These five models are namely a) Bernice, b) TwHIN-BERT, c) MuRIL, d) mBERT, and e) XLM-RoBERTa. It is found that our test-time ensemble technique solves this multi-class sentiment classification problem defined over the proposed dataset in a better way as compared to individual transformer models. Encouraging experimental outcomes confirms the efficacy of the proposed approach
Rupam Bhattacharyya, "Classification of Multilingual Financial Tweets Using an Ensemble Approach Driven by Transformers", International Journal of Information Engineering and Electronic Business(IJIEEB), Vol.17, No.2, pp. 51-67, 2025. DOI:10.5815/ijieeb.2025.02.02
[1]Tuan Hao Ly and Khanh Nguyen. Do words matter: Predicting ipo performance from prospectus sentiment. In 2020 IEEE 14th International Conference on Semantic Computing (ICSC), pages 307–310. IEEE, 2020.
[2]Elena Fedorova, Sergei Druchok, and Pavel Drogovoz. Impact of news sentiment and topics on ipo underpricing: Us evidence. International Journal of Accounting & Information Management, 2021.
[3]Jim Kyung-Soo Liew and Garrett Zhengyuan Wang. Twitter sentiment and ipo performance: A cross-sectional examination. Journal of Portfolio Management, 42(4):129, 2016.
[4]Erkut Memi¸s, Hilal Akarkamçı, Mustafa Yeniad, Javad Rahebi, and Jose Manuel Lopez-Guede. Comparative study for sentiment analysis of financial tweets with deep learning methods. Applied Sciences, 14(2):588, 2024.
[5]Carol Myers-Scotton. Duelling languages: Grammatical structure in codeswitching. Oxford University Press, 1997.
[6]Hatem Haddad, Ahmed Cheikh Rouhou, Abir Messaoudi, Abir Korched, Chayma Fourati, Amel Sellami, Moez Ben HajHmida, and Faten Ghriss. Tunbert: Pretraining BERT for tunisian dialect understanding. SN Comput. Sci., 4(2):194, 2023.
[7]Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015.
[8]Rupam Bhattacharyya, Zubin Bhuyan, and Shyamanta M Hazarika. Inferring semantic object affordances from videos. In Computer Vision and Image Processing: 5th International Conference, CVIP 2020, Prayagraj, India, December 4-6, 2020, Revised Selected Papers, Part III 5, pages 278–290. Springer, 2021.
[9]Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
[10]Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
[11]Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark Dredze. Bernice: A multilingual pre-trained encoder for twitter. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6191–6205, 2022.
[12]Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, and Ahmed El-Kishky. Twhin-bert: A socially-enriched pre-trained language model for multilingual tweet representations. arXiv preprintarXiv:2209.07562, 2022.
[13]Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, et al. Muril: Multilingual representations for indian languages. arXiv preprint arXiv:2103.10730, 2021.
[14]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[15]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.
[16]Gopal Acharya and Dushyant Mahadik. Underpricing effect in initial public offerings: A systematic review and bibliometrics analysis. Available at SSRN 4541672, 2023.
[17]Kelvin Du, Frank Xing, Rui Mao, and Erik Cambria. Financial sentiment analysis: Techniques and applications. ACM Computing Surveys, 56(9):1–42, 2024.
[18]Yu Ma, Rui Mao, Qika Lin, Peng Wu, and Erik Cambria. Multi-source aggregated classification for stock price movement prediction. Information Fusion, 91:515–528, 2023.
[19]Pyry Takala, Pekka Malo, Ankur Sinha, and Oskar Ahlgren. Gold-standard for topic-specific sentiment analysis of economic texts. In LREC, volume 2014, pages 2152–2157, 2014.
[20]Ali Albada, Muataz Salam Al-Daweri, Rabie A Ramadan, Khalid Al Qatiti, Li Haoyang, and Peng Shutong. Determinates of investor opinion gap around ipos: A machine learning approach. Intelligent Systems with Applications, page 200420, 2024.
[21]Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. Improving bert fine-tuning via self-ensemble and selfdistillation. arXiv preprint arXiv:2002.10345, 2020.
[22]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019.
[23]Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models. arxiv 2019. arXiv preprint arXiv:1908.10063, 2019.
[24]Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019.
[25]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
[26]Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages. In Findings of EMNLP, 2020.
[27]Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. Bertweet: A pre-trained language model for english tweets. arXiv preprint arXiv:2005.10200, 2020.
[28]Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258–266, 2022.
[29]Francesca Cornelli, David Goldreich, and Alexander Ljungqvist. Investor sentiment and pre-ipo markets. The journal of finance, 61(3):1187–1216, 2006.
[30]Luca Gregori Gian, Camilla Mazzoli Luca, Marinelli, and Severini Sabrina. The social side of ipos: Twitter sentiment and investorsâC™ attention in the ipo primary market. African Journal of Business Management, 14(12):529–539, 2020.
[31]Meera Mehta, Shivani Arora, Shikha Gupta, Arun Jhulka, et al. Social listening through sentiment analysis of twitter data: a case study of paytm ipo. SocioEconomic Challenges, 6(3):39–47, 2022.
[32]Weiwei Jiang. Applications of deep learning in stock market prediction: recent progress. Expert Systems with Applications, 184:115537, 2021.
[33]Ross Gruetzemacher and David Paradice. Deep transfer learning & beyond: Transformer language models in information systems research. ACM Computing Surveys (CSUR), 54(10s):1–35, 2022.
[34]Sayyida Tabinda Kokab, Sohail Asghar, and Shehneela Naz. Transformer-based deep learning models for the sentiment analysis of social media data. Array, page 100157, 2022.
[35]Aysun Bozanta, Sabrina Angco, Mucahit Cevik, and Ayse Basar. Sentiment analysis of stocktwits using transformer models. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1253–1258. IEEE, 2021.
[36]Zekeriya Anil Guven. Comparison of bert models and machine learning methods for sentiment analysis on turkish tweets. In 2021 6th International Conference on Computer Science and Engineering (UBMK), pages 98–101. IEEE, 2021.
[37]Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. Fine-grained analysis of financial tweets. In Companion Proceedings of the The Web Conference 2018, pages 1943–1949, 2018.
[38]Aapo Pietiläinen and Shaoxiong Ji. Aaltonlp at semeval-2022 task 11: Ensembling task-adaptive pretrained transformers for multilingual complex ner. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1477–1482, 2022.
[39]Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249, 2018.
[40]Hongzhi Zhang and M Omair Shafiq. Survey of transformers and towards ensemble learning using transformers for natural language processing. Journal of big Data, 11(1):25, 2024.
[41]Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821–10836, 2022.
[42]Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7959–7971, 2022.
[43]George Manias, Argyro Mavrogiorgou, Athanasios Kiourtis, Chrysostomos Symvoulidis, and Dimosthenis Kyriazis. Multilingual text categorization and sentiment analysis: a comparative analysis of the utilization of multilingual approaches for classifying twitter data. Neural Computing and Applications, 35(29):21415–21431, 2023.
[44]Ziyun Zhou, Hong Huang, and Binhao Fang. Application of weighted cross-entropy loss function in intrusion detection. Journal of Computer and Communications, 9(11):1–21, 2021.
[45]Yaoshiang Ho and Samuel Wookey. The real-world-weight cross-entropy loss function: Modeling the costs of mislabeling. IEEE access, 8:4806–4813, 2019.
[46]Artittayapron Rojarath and Wararat Songpan. Cost-sensitive probability for weighted voting in an ensemble model for multi-class classification problems. Applied Intelligence, 51:4908–4932, 2021.
[47]Léon Bottou. Stochastic gradient descent tricks. Neural Networks: Tricks of the Trade: Second Edition, pages 421–436, 2012.
[48]Mohammad Hossin and Md Nasir Sulaiman. A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1, 2015.
[49]Radu Stefan, George Carutasu, and Marian Mocan. Ethical considerations in the implementation and usage of large language models.