Accuracy Evaluation of ChatGPT 4.0 in Text Annotation of Turkish Islamic Literature

Abstract

This study investigates the application of artificial intelligence, specifically Chat-GPT-4.0, in the field of Turkish Islamic literature with a focus on text annotation and accuracy evaluation. With advancements in machine learning and deep learning algorithms, AI has been utilized across various fields, from education to healthcare, enhancing the depth and speed of analyses traditionally done manually. In this study, ChatGPT-4.0 was employed to interpret a Turkish qasida-a form of classical panegyric poetry dedicated to Sultan Selim I. The research evaluated twenty couplets from the qasida, split into two groups: the first ten were analyzed with directed prompts that focused on the relationship between music and emotion, while the remaining ten were interpreted without guidance, allowing the AIto provide autonomous commentary. The first section of the study focused on a structured approach where specific prompts were used to guide ChatGPT-4.0 in exploring themes of musical symbolism and emotional depth. This guided analysis included topics like "Music and Emotional Expression," "Symbolism of Mehter Music," and "The Emotional Role of the Ney (reed flute) in Ottoman Poetic Traditions." By drawing connections between these themes and the couplets, the AI was encouraged to explore the symbolic associations between certain musical instruments and emotional states. The ney, for example, was interpreted as an instrument conveying melancholy, aligning with its cultural significance in Sufi music. This directed approach led to highly consistent results, with the responses scoring well in terms of factual accuracy, thematic relevance, and clarity. In the second part of the analysis, ChatGPT-4.0 was tasked with interpreting the couplets without directed prompts, allowing for a broader analysis not constrained to any specific themes. Here, the AI adopted a more conceptual approach, exploring general cultural, aesthetic, and emotional aspects rather than focusing solely on musical connections. This autonomy led to varied interpretations that were not always aligned with the intended themes but introduced additional insights. Interestingly, these responses scored higher in clarity and specificity but showed slightly lower scores in factual accuracy and evidence-based support compared to the directed analysis. The results suggest that while AI can independently provide insightful interpretations, directed prompts are beneficial for maintaining thematic focus and analytical consistency in specialized fields like Turkish Islamic literature. Comparing the results from both sections, the study identified distinct strengths and limitations in AI-driven text annotation. The guided analysis underscored the value of prompt engineering for obtaining focused, factually consistent insights, particularly in examining the complex cultural and emotional dimensions of Turkish Islamic literature. For instance, by structuring questions around musical themes, the AI could more effectively generate contextually accurate interpretations. Conversely, the unguided responses illustrated ChatGPT-4.0's potential for producing broader cultural reflections, allowing the model to interpret the couplets through a more diverse lens, though sometimes at the expense of thematic precision. An important aspect of this study was the accuracy evaluation of the AI's responses. For this purpose, a six-criterion framework-Accuracy of Information Index (AOI)-was developed, rating responses on a scale from 0 to 5 based on the following factors: factual accuracy, evidence-based support, clarity, thematic relevance, specificity, and consistency. The results revealed that the guided approach consistently achieved higher scores in factual accuracy, evidence support, and thematic relevance, demonstrating the importance of structured guidance in AI-assisted literary analysis. On the other hand, the unguided section achieved higher clarity and specificity scores, indicating that, while the AI provided more autonomous insights, some interpretations deviated from the intended themes. The study's methodology also included a detailed comparative analysis of sources cited in each section. In the guided analysis, references to Ottoman music and poetry sources, such as works by Turkish musicologists and historians, were integrated into the AI responses to enhance reliability. Citations were provided in APA style, further validating the AI's interpretations. In contrast, the unguided analysis drew more general connections, citing broader literary and historical sources, which, while insightful, occasionally lacked the thematic depth of the guided responses. The research provides a comprehensive overview of how AI, when combined with structured prompts, can effectively contribute to the study of complex literary texts, offering both thematic insights and cultural analysis. This study suggests that artificial intelligence can act as a valuable assistant in Turkish Islamic literature by supporting traditional methods with a systematic, technology-driven approach. However, the findings also caution against over-reliance on AI, as accuracy and thematic relevance can be compromised without proper guidance. For future research, integrating more diverse prompts or refining prompt strategies could enhance the precision and depth of AI-generated insights, enabling scholars to explore Turkish Islamic literature from new perspectives. Ultimately, this study contributes to ongoing discussions on AI's role in the humanities, specifically in analyzing culturally rich texts within Turkish Islamic literature. The research underscores the potential of AI to support and enrich traditional interpretative methods, suggesting that, with the proper guidance, AI tools can aid scholars in making literary works more accessible and interpretable. While AI may not replace human scholarship, it can significantly complement it, allowing researchers to uncover nuanced layers of meaning within classical literature.