Pitfalls of Chat User Interfaces (not only) for Data Analytics

Chatting with AI is becoming very popular across many domains. And data analytics is no exception. But entering a simple text prompt into a chat window and getting back a polished gem of insight is still a distant future. Let's explore some of the pitfalls of using AI chat interfaces (not only) for data analytics and how to avoid them.

A new UI paradigm emerged after more than 50 years

Ever since most of today's users started using computers, the user interface hasn't changed much. Command-based interfaces have been around for more than 50 years. The user and computer are simply taking turns. It all started with a command line interface and then moved to a graphical user interface. The user submits a command or clicks on a button, and the computer fulfills the request in a more or less predictable way. The shapes of buttons changed over time, but the basic principle prevailed. The request and the result, approaching the end result one command at a time.

However, a new paradigm has emerged with the advent of ChatGPT, Midjourney, and other AI-powered chat tools. Jacob Nielsen labeled this paradigm shift as Intent-Based Outcome Specification. It means that the user describes the desired outcome in full (called prompt), and the computer does its best to understand the user's wishes and generates the output. But except for this initial prompt, the user has no control over the creation process. To the end user, the AI system is a black box.

Users are used to clicking around and trying what works

Let's now imagine an ordinary user. For example, you. How did you manage to learn to use some new app recently? If you are like most users and you faithfully ignore manuals and tutorials, the chances are that you just opened the app and tried to click around and tried all the possibilities. The trial-and-error approach, one button at a time. And before long, you probably managed to handle the app just fine. But now, there are two big problems ahead:

1. There’s typically only a single button in the AI chat interfaces, which is usually disabled until the user fills in the prompt.
2. The whole “error” part from the “trial-and-error” is missing. Since the current LLMs are generative AIs, they generate text (or images) and rarely produce an actual error. And it’s up to the user to tell whether the AI is hallucinating or not.

Let’s now explore the first problem in more detail.

How do you click around when all you have is a chat window?

Have you ever tried to write a short story? An article? Or even a book? Can you recall that feeling before you put pen to paper for the first time, or you were staring into the empty white chasm of a blank page?

A similar feeling can induce a simple AI chat interface in the users. Staring at a blinking cursor in the ChatGPT window and having no idea what should be written first could be paralyzing.

You have probably seen a variation of this meme with designers/developers/creatives and other professions:

In order to replace [insert job title], clients will need to accurately describe what they want.

We're safe, people.

It's funny because it's true. The essence of those jobs is often to translate the clients' vague dreams into the designs/code/visuals.

So, it should not be surprising that transforming ideas into words is hard, and most likely, more than half of the population cannot articulate ideas in a written form sufficiently well to write usable prompts for AI chat systems. They can only scratch the surface of the possibilities of AI systems without the ability to utilize their full potential.

A good indication of this problem is companies hiring prompt engineers. This raises a major red flag in the current state of the usability of generative AI systems.

What general steps can we take to make it easier for the users?

Take it easy. AI chat interfaces are not going away anytime soon, so what would be the way to utilize them without confusing users?

Do not throw users into the deep water right away. Avoid an all-in approach. Simply putting an AI chat interface in your product won't cut it. You have to provide other means of control, not just a chat window. Or, the other way around, add an AI chat interface to your product as a complementary feature - not replacing the current functionality.

Here are some general steps to take:

• User-centered design: Build and test the AI chat interface with less skilled users in mind. Gather feedback from diverse user groups and iterate on the design.
• Clear guidance and discoverability: Offer clear instructions and tutorials to guide users through the AI chat interface. An introductory onboarding process can familiarize users with the system's capabilities and limitations and allow them to discover what could be done.
• Performance: Processing long prompts requires a lot of time, which negatively impacts user experience. Spend extra time polishing the states when the AI processes the prompt - the ones where the users have to wait.
• Contextual awareness: Design the AI to provide assistance where users expect it and pre-fill in the right information to make it easier to start writing the prompt. During the chat with AI, remember previous interactions within a session, allowing the system to provide more context-aware responses.
• Error handling and feedback: Design the AI to handle misunderstandings gracefully. If the AI can't comprehend a request, it should ask the user to rephrase the request or offer examples of valid questions.
• Preventing hallucinations: AI LLMs tend to hallucinate without notice. It is quite hard to detect the behavior since the AI generates the result - and then it’s up to the user to tell whether it is true or false.
• Transparency: Clearly communicate to users that they are interacting with an AI. Inform users about how their data is used and the steps taken to ensure privacy.
• Limit information overload: Offer concise responses with the option for users to request more detailed information.
• Feedback loop: Implement a feedback mechanism that lets users rate the accuracy and helpfulness of the AI's responses, allowing for continuous improvement.

What UI elements can help users work well with AI chat interfaces?

By integrating these UI elements, AI chat interfaces can be more user-friendly and intuitive, catering to both novice and experienced users while offering them greater control and understanding of their interaction with the chat interface:

• Suggested and example prompts to overcome the problem of the empty chat box: Suggested prompts provide a starting point, guiding users on potential questions or actions. Display a few suggested questions or commands in faded text within the AI chat input box, or list them above or below. This reduces the "analysis paralysis" that might occur when presented with an empty interface and gives users an idea of the chatbot’s capabilities.
• What other users are looking for: By showcasing popular or trending queries, users can get insights into what others find valuable, which might also be relevant to them. New or uncertain users can gain inspiration from these queries, broadening their understanding of the chatbot's capabilities and potential applications.

• Past prompts: Users often have recurring needs or may wish to revisit previous analyses. Showing past prompts allows for easy repetition of common queries. Clicking on a past prompt would allow the user to rerun that query. This saves time and reduces the need for repetitive typing, especially for complex queries.
• UI elements to control the results within the chat: User interaction doesn't need to be limited to the chat window - it can be accompanied by traditional buttons users are more familiar with:
• • Change the chart type: Different users may have preferences for how data is visualized, or certain data might be better represented with specific chart types. Offer icons or dropdown menus adjacent to visual results that allow users to switch between chart types (e.g., from bar to column or line charts).
  • Show underlying data: While visual representations are insightful, sometimes users might need to view the raw data or numbers behind those visualizations. Provide a "View Data" button or toggle. When activated, the system displays the data table or values that the chart is based on. This provides more transparency and allows users to make detailed verifications when needed.
  • Show the answer’s source: Showing details of the query the chatbot used to arrive at its answer should help users understand where the data is coming from and build trust in the chatbot’s capabilities. It also allows users to double-check that the chatbot understands their questions correctly.

Blending the Old with the New: Making AI Chats Intuitive

AI-powered chat interfaces offer new opportunities in data analytics but can be daunting, especially for less experienced users. Blending traditional UI elements with an AI chat interface can make the experience more intuitive and productive, combining the best of both worlds.

You can also check how we approach generative AI in GoodData in an excellent article by Patrik Braborec.