Chat GPT Heuristic Evaluation
Abstract
Chat GPT is an AI chatbot that understands and generates human-like text, in addition to generating pictures, searching the internet, and more. This evaluation uses the Nielsen-Molich design heuristic and a custom accessibility heuristic to rate the usability and accessibility of Chat GPT. This report will identify Chat GPT’s shortcomings concerning Accessibility and Usability, as well as recommending ways Chat GPT can improve. Chat GPT’s developers can use this information to make their product easier to use for a wider range of people.
Introduction
Chat GPT is a generative AI chatbot developed by Open AI. It can understand and generate human-like text, allowing it to hold conversations like a human. People use Chat GPT to answer questions, write for them, summarize text, generate pictures, and much more. Chat GPT targets many different audiences and is used by students, office workers, content creators, and many more groups for both productivity and recreation.
Purpose
This evaluation aims to assess Chat GPT in terms of accessibility and usability, and to provide recommendations about improving Chat GPT. The results of this evaluation can make Chat GPT more effective and available to disabled persons. Additionally, the results of this evaluation may assist in making Chat GPT easier and more enjoyable to use for everybody.
Methodology
This evaluation will use the Nielsen-Molich design heuristics and a custom set of accessibility heuristics. I will also use Apple’s built in screen reader, Chrome Lens, and Lighthouse. I will conduct this evaluation from the perspective of a college student who has used technology most of their life.
Findings
Design Heuristic
Visibility of System Status
Chat GPT uses dots and real time text generation to keep users informed about its status. Additionally, it shows users when it is thinking, searching the internet, generating a photo, or doing anything else.
The picture above shows the dot Chat GPT displays when starting a response.
The image above shows Chat GPT's live text generation
Match Between System and Real World
Chat GPT generates and understands human-like text, so interacting with it feels like conversing with another person. Chat GPT also learns from and memorizes things about the user, like another person would.
User Control and Freedom
Chat GPT allows users to stop responses, edit previous prompts, and regenerate prompts. Unfortunately, I was unable to resume responses that I stopped, and I was unable to find the regenerate response button. Chat GPT boasts a collection of Open AI and user made Chat GPTs that are suitable for specific tasks. Additionally, users can customize their base Chat GPTs personality and give it information about themselves to improve interactions. Users can easily switch between versions of Chat GPT and may enable it to search the internet or think longer before responding.
The picture above shows the stop response button, in the red circle.
The picture above shows buttons that give users more freedom while using Chat GPT
Consistency and Standards
Chat GPT follows web standards. The sidebar, help, and settings/customization remain in their usual positions. Furthermore, the layout is similar to Google’s, with everything centered around the main chat bar.
Error Prevention
Chat GPT does not cause many technical errors, but its responses are not always correct. Using Chat GPT’s responses without checking them can cause problems for users.
Recognition over Recall
Past conversations are saved by Chat GPT unless the user starts a temporary conversation, allowing users to look back at most of their conversations with Chat GPT. Chat GPT also remembers things from earlier conversations and about the user. Most of the icons are easily understood, but I was confused by the icon for switching the version of Chat GPT you are using, as I thought it was the regenerate response button.
The image on the right shows Chat GPT's side bar and saved chats
Flexibility and Efficiency
While Chat GPT offers flexibility for chat interactions, users can only access most parts of the website one way. However, Chat GPT does allow both text and voice interactions.
The picture to the right shows Chat CPT’s voice mode
Aesthetic and Minimalist Design
The design of Chat GPT is sleek and minimalist. Its black and white color scheme of mostly open space, limits distractions. Chat GPT centers around the chat bar, with all other elements bordering the screen. This makes it easy to get right into chatting and minimizes mistakes.
Error Recognition and Recovery
Chat GPT warns the user about potential errors, especially with its unreliable responses. AI will also help users recognize and fix errors.
Help and Documentation
Users can find help pages on the bottom right corner of the screen. Chat GPT also offers help to users if they ask for it.
Accessibility Heuristic
Layout
Screen readers navigate Chat GPT well, and all of its elements remain on screen when the screen size changes.
Design Elements
Chat GPT uses common HTML elements including a sidebar for navigation that can be hidden or exposed in the top left, help button in the bottom right, profile and settings in the top right, and the main chat bar in the middle. The Global section contains a search bar and featured items, with other Chat GPTs being sorted by category and popularity.
The picture above displays the Chat GPT global section which has Chat GPTs made to assist with specific tasks
Color
Chat GPT uses a simple black, white, and grey color screen with an abundance of open space and no distractions.
Writing
Chat GPT writes in a human-like and easily understandable way. Additionally, it learns from users and adjusts responses to suit the user’s needs.
Font and Text
Unfortunately Chat GPT only provides one font and text size for users, but the text is readable for most people.
Text Organization
Chat GPT typically uses spacing to organize responses, but I had to ask it to organize a response better for it to use bold text and indentations in its response.
The picture above shows Chat GPT's basic text formatting
The picture above shows Chat GPT's formatting after I asked it to improve the basic formatting
Buttons and Links
When using a mouse, buttons and links are easy to notice. They are highlighted when users hover over them, and many buttons include descriptions when hovered over. Buttons are surrounded by noticeable borders, making it clear what they are.
Images
Chat GPT does not include alt text for images it creates, but it does include descriptions of them in the same response.
Media
Chat GPT generates photos but it is unable to make videos. Although Chat GPT cannot make videos, it can recommend videos it finds online. Chat GPT’s text responses can be read aloud and its voice mode allows users to interact with it by talking and listening.
Navigation
The sidebar allows users to navigate Chat GPT easily, and users are never required to memorize things from other pages. Unfortunately, there is only one way to access most of the site's features and some of the buttons are tightly spaced.
Forms
Captcha is not necessary to access Chat GPT and there are few forms for users to fill out. Chat GPT identifies and corrects spelling and grammatical errors. Additionally, when it is confused by a prompt, it will ask the user to clarify.
Keyboard Navigation
Keyboard navigation is a major weakness for Chat GPT, as it does not highlight links when you hover over them and I had to hit tab twice before being able to hit enter on some buttons.
Recommendations
High Priority Issues
-
Chat GPT requires major improvement in keyboard navigation. I could not always discern where I was, especially with past chats. Past chats do not visually change when hovered over during keyboard navigation.
-
The issues with resuming stopped chats and not being able to find the regenerate button hurt the flexibility of Chat GPT. I remember being able to regenerate responses in the past, so I was disappointed that I could not do it anymore.
Medium Priority Issues
-
There is only one way to get to many parts of the Chat GPT website. Creating more ways for users to navigate the site would give them more freedom and enjoyment using Chat GPT.
-
The ability to change fonts and text size could make the text easier to read for some users.
Low Priority Issues
-
Adding a dark mode would increase Chat GPT’s flexibility and make it more enjoyable for some users.
Conclusions
Overall, Chat GPT proves usable and accessible, but it could improve its keyboard navigation, customer freedom with chat responses, adding multiple ways to get to sections of the site, and other small elements. I found the core of Chat GPT very easy and satisfying to use, and it can do so much in so many different ways. Chat GPT is an incredibly powerful tool that will only add more functionalities, so Open AI should prioritize accessibility and usability going forward.