JISARA

Abstract: With the widespread use of Large Language Models (LLMs) in various applications, there has been growing interest in leveraging their capabilities to improve user experiences and streamline processes. However, given the availability of various LLMs and Retrieval-Augmented Generation (RAG) systems, it is crucial to understand the differences between these technologies to effectively implement them and maximize their potential for providing better services.  This paper investigates whether a RAG-empowered mobile app can enhance user experiences by delivering more relevant responses to specific user inquiries than a system equipped solely with an LLM. We focus on RAG, facilitated by LangChain, and compare its effectiveness to that of LLMs. As a demo case for comparison, we developed a chatbot-based internship placement system using React Native, integrating the ChatGPT API and LangChain for personalized, relevant responses. By evaluating chatbot responses using the RAG Assessment (Ragas) framework with metrics such as context precision, context recall, faithfulness, and answer relevancy to measure the quality of the RAG pipeline, we found that the RAG-empowered system consistently delivered more context-specific answers. A qualitative comparison revealed that the LLM system produced more generic responses than the RAG system. RAG systems can enhance the efficiency and effectiveness of internship placements by offering tailored assistance. Our findings highlight the potential of advanced NLP technologies to revolutionize applications such as chatbots, promoting innovation and enhancing user experiences.

Download this article: JISARA - V18 N4 Page 4.pdf

Recommended Citation: Kise, S., Chung, S., (2025). User Experiences in a RAG-Empowered Application. Journal of Information Systems Applied Research and Analytics 18(4) pp 4-13. https://doi.org/10.62273/WQTL4388