Evaluating interactive bibliographic information retrieval systems: A user-centered approach

As an integral part of research activities, scholars frequently interact with bibliographic information retrieval systems to acquire scholarly information. With the dramatic increase of scientific literature, there is the pressing need of building effective and efficient bibliographic information retrieval systems that support more granular and complex bibliographic information needs. Investigating users' interaction with these systems plays an important role in understanding the systems' features and usability issues, which contributes to better system design and implementation. In this paper, we evaluate three interactive bibliographic information retrieval systems proposed in our previous studies using a user-centered approach, including form-, natural language-, and visual graph-based systems. This study recruits 20 participants to evaluate the three systems from the aspects of success rate, search time, query size, usefulness, ease of use, ease of learning, and satisfaction. Results showed that: the form-based system needed the least time to formulate simple bibliographic queries and allowed the participants to make the least mistakes; the natural language-based system needed the least time to formulate high-complexity bibliographic queries and was rated as the most useful, easy-to-use, and easy-to-learn system; and the visual graph-based system's strengths lie in the support of complex queries as shown that it got better ratings of usefulness and satisfaction by the high-complexity task group.