Multilabel Aspect-Based Emotion Analysis Pada Ulasan Aplikasi IKD: Pengaruh Focal Loss dan Threshold Tuning Menggunakan Indobert

Abstract

User reviews of the Identitas Kependudukan Digital (IKD) application contain various emotions toward different service aspects. These reviews not only reflect the level of service satisfaction but also encompass user experiences, complaints, expectations, and public perceptions regarding the quality of the system. This study aims to develop a multi-label Aspect-Based Emotion Analysis (ABEA) model using an end-to-end IndoBERT architecture to identify user emotions across each service aspect of the IKD application. Additionally, it analyzes the impact of implementing Focal Loss and threshold tuning on classification performance under highly imbalanced label distributions. Data were collected from 13,197 user reviews on the Google Play Store spanning from June 2024 to November 2025 using web scraping methods, which were subsequently cleaned and filtered to yield 6,891 data entries. Service aspects were empirically identified using BERTopics. Labeling was conducted by three human annotators and two AI annotators, with the final labels determined through majority voting. The model was developed across 6 experimental scenarios varying in preprocessing, Focal Loss, threshold tuning, and data split ratios. Evaluation was performed using F1 Score Macro, F1 Score Micro, Precision, Recall, and Hamming Loss metrics. BERTopic achieved a Coherence Score of 0.6196 and a Topic Diversity of 0.92 with 5 representative aspects. The most optimal model was obtained using a configuration of Focal Loss, a threshold of 0.4, and a 60:20:20 split ratio, achieving an F1 Score Macro of 0.3916, a 24.1% increase from the baseline, alongside an F1 Score Micro of 0.9134 and a Recall of 0.9423. The selected model was successfully integrated into a web-based system using the Flask framework to visualize the classification results. Anger dominated the reviews concerning the Login & Akses Akun and Scan Barcode ke Dukcapil aspects, whereas the Dokumen & Layanan Digital aspect recorded the highest joy emotion. The combination of Focal Loss and threshold tuning proved effective in handling imbalanced label distributions in Indonesian multi-label ABEA classification.