NAVER
Full-time
2023-24
AI-Generated Replies
Role
Product Designer
Duration
8 months
(Oct 2023-Jun 2024)
Collaboration
TImeline & skills
Launched a closed-beta version in April 2024
How to save time and effort replying
to customer reviews?
✸
Problem
Writing creative replies is time-consuming.
Managing malicious reviews is highly challenging.
✸
Impact
A beta survey of 1,000 users showed 97% satisfaction.
Most of replies from business users are just copy-paste.
curiosity 1.
What pain points exist in the current reply-writing experience?
curiosity 2.
How can I encourage them to leave more genuine and effective replies?
To define the problems, I conducted a 5-minute survey with 200 business users and 45-minute interviews with 5 users across various industries.
1
Business users spend an excessive amount of time to reply.
6 hours a week on average
5 minutes per 1 review
3 times a week
2 hours for each time
Smart Place user survey (2023)
2
They often struggle with the pressure of crafting appropriate replies.
Smart Place user in-depth interview (2023)
3
They have difficulties reporting malicious reviews.
Unsure of the correct process for submitting a report.
Uncertain whether it is appropriate to file a report.
Small Enterprise and Market Service (2023), <Restaurant Enterprises' Review Platform Usage Survey>
Re-generate replies
Providing human-like AI replies,
not like repeating similar answers like a parrot.
Designing an intuitive and easy-to-learn UI
for a newly introduced feature.
Distinguishing between reviews to be reported
and those to be cautious of.
Setting up the review sentiment level system.
Generative AI should write appropriate responses correlating to the actual review, whether it is positive or negative.
Subdivision in negative level.
To make the response more detailed, I've decided to split the negative level into two sub-levels.
If a review isn't severe enough to warrant immediate suspension but could offend staff, I labeled it as 'Intense Negative.'
For a review that is fair but is more negative and needs quick attention, I classified it as 'Mild Negative.'
Labeling the specific examples to make a dataset.
Our team brought a set of specific review examples and discussed whether the comment was neutral or negative, mild or intense negative, and intense negative or malicious.
Do you think the keyword "old" is neutral or negative?
We discussed whether the reviews containing the word "old" should be classified as negative or neutral. Some thought "old" carries a negative connotation, while others argued that it is a neutral statement of fact. As a result, we decided that it would be considered neutral unless stronger phrases such as "worn out" or "too old" were used.
Designing an intuitive interface
Intuitive interface that enables first-time users to be easily engaged.
AI replies are a new feature. Instead of offering detailed controls, I focused on designing an interface that first-time users could intuitively understand.
Test 1.
The 3 most important things the user should notice clearly:
Section
content
Buttons
Interface Design Tests
I initially tried to maintain the existing layouts. However, it became too complicated to recognize the tasks in the order of importance.
Therefore, I reorganized some features on the back burner into the kebab menu to make it concise.
In addition, I avoided using words like 'generate,' which is a technology-centered jargon but not user-centered.
Test 2.
To recognize each reply as an option users can choose, I utilized a tangible square button rather than pagination components.
Component Design Test
Observing users through a usability test.
Check if the user can handle the product as intended in the scenario I've designed.
Before launching the beta, I led a quick usability test using a prototype by Figma.
I organized 4 different scenarios, which is the most basic operating scenario, to check for major defects in the basic controls.
What I found.
The main misunderstanding is with the 'Re-generate' function. Initially labeled 'See other replies,' users thought it would reset the current reply, so they hesitated to click it.
I then changed the label to 'See More Options,' aiming to show that the current reply would remain, with additional choices available.
A small change makes a big difference.
To see whether it works as I intended, I conducted a usability test again. I also recruited 4 other business users.
In the first user test, 3 out of 4 users didn't understand the button's function. However, with just a slight change in the label, all 4 users in the second test understood its meaning as I intended.
Presenting AI
as a personified character.
To better understand the process of AI writing for users, personified branding is highly effective. Therefore, I designed the character and created character design guides.
I added small character badges to AI-generated responses, invisible to customers, allowing owners to identify which reviews received assistance from the AI.
Becoming a flexible designer who can lead
the end-to-end design process.
Designing a new product from start to finish was both challenging and rewarding. From initial research to final implementation, I navigated various stages, ensuring a user-centered approach throughout.
A key takeaway was the importance of flexibility. User feedback and technical constraints required design adjustments during the project. By staying adaptable and communicating openly with cross-functional teams, I ensured the final product met business goals and delivered real user value.
I resigned my company to pursue Master of Design program at UC Berkeley. However, my teammates are currently working hard toward the next launch. I fully support them and am excited to see the next steps of our progress!
© Jeongmin Lee 2024
