JISARA

Abstract: Large language models (LLMs) and the use of artificial intelligence (AI) are becoming increasingly integrated into everyday life. As the models continue their training and improvement, leading technology companies are competing to create the most advanced artificial assistant. This competition has produced a variety of large language models, each with different capabilities. While LLMs like ChatGPT, Claude, and Gemini have proven effective at handling everyday tasks, they often lack the domain-specific expertise necessary for more specialized consultations. This limitation makes it essential to integrate a targeted knowledge base when developing chatbots for specific domains. Therefore, this study aims to address the question: Is there a model that clearly outperforms the others, considering correctness and comprehensiveness? In the study, we conducted an experiment to investigate the performance of chatbots utilizing LLMs within the context of a custom, mid-scale knowledge base designed for a university located in the southeastern United States. Using a web-crawled knowledge base, we created chatbots across multiple platforms and LLMs, testing them against a set of predefined questions to evaluate correctness and comprehensiveness. The findings highlight the disparity in the capability of LLMs and offer practical guidance for their effective use.

Download this article: JISARA - V19 N2 Page 11.pdf

Recommended Citation: Villeda Roblero, A., Song, Y., Gebauer, J., Shi, Y., (2026). A Study of Chatbots Performance with Large Language Models in Scenario of a Higher Education Institution. Journal of Information Systems Applied Research and Analytics 19(2) pp 11-24. https://doi.org/