Skip to main navigation Skip to search Skip to main content

Can LLMs See What I See? A Study on Five Prompt Engineering Techniques for Evaluating UX on a Shopping Site

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Usability testing is essential for improving digital user experiences but has practical limitations in terms of cost-effectiveness. Recent advancements in multimodal Large Language Models (LLMs), like ChatGPT-4, offer new possibilities for UX evaluations. This study investigated the most effective prompt engineering techniques for identifying UX issues in digital interfaces. To achieve this, five prompt engineering techniques were carefully selected based on previous research, and the outputs generated using these techniques were analyzed based on severity assessment criteria. We discovered that Role Prompting and (Zero-Shot) Chain of Thought Prompting were highly effective. Further investigation revealed that a hybrid approach combining both techniques produced the best results. Our findings shed light on the possibility of using multimodal LLM as a UX evaluator, offering meaningful value for future advancements in LLM-based UX evaluations.

Original languageEnglish
Title of host publicationCHI EA 2025 - Extended Abstracts of the 2025 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9798400713958
DOIs
Publication statusPublished - 2025 Apr 26
Event2025 CHI Conference on Human Factors in Computing Systems, CHI EA 2025 - Yokohama, Japan
Duration: 2025 Apr 26 → 2025 May 1

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2025 CHI Conference on Human Factors in Computing Systems, CHI EA 2025
Country/TerritoryJapan
CityYokohama
Period25/4/26 → 25/5/1

Bibliographical note

Publisher Copyright:
© 2025 Copyright held by the owner/author(s).

All Science Journal Classification (ASJC) codes

  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design
  • Software

Fingerprint

Dive into the research topics of 'Can LLMs See What I See? A Study on Five Prompt Engineering Techniques for Evaluating UX on a Shopping Site'. Together they form a unique fingerprint.

Cite this