| Human_Evaluation,Order,used_ai_for_emotions,Details,Rag_No_of_Turns,Rag_Helpfulness,Rag_Supportive,Rag_Clarity,Rag_Groundedness,Rag_Overall,Rag_Comments,Non_Rag_No_of_Turns,Non_Rag_Helpfulness,Non_Rag_Supportive,Non_Rag_Clarity,Non_Rag_Groundedness,Non_Rag_Overall,Non_Rag_Comments | |
| Participant_1,nonrag -> rag,No,,8,3,3,3,3,3,The second model was slower but had comparatively better response,8,2,3,4,3,3,Was quicker in response | |
| Participant_2,rag -> nonrag,Yes,Chat gpt. Used very rarely,6,5,4,4,3,3,The color combination making it very difficult to read the stuff on the screen. \nAnd the response time is very slow.,6,3,3,4,4,4, | |
| Participant_3 ,nonrag -> rag,No,,5,5,4,5,5,4,"This bot was very organized and gave on point answers. However, the tone was very direct. It would have been a great overall experience if the tone was slightly more warm and friendly.",5,4,5,4,5,4,I feel the response can be more clear in terms of organization. Some prompts can be answered in bullet points rather than one huge paragraph. | |
| Participant_4,nonrag -> rag,No,,7,4,4,4,2,4,This one felt a bit more rebellious and did not seem to agree with what I said and was a lot more opinionated.,12,5,4,2,4,4,Does not have very clear opinions in difficult situations and tries to play it safe. | |
| Participant_5,rag -> non-rag,No,,4,4,4,4,4,4,this was more emotional oriented then solution driven ,4,5,5,4,5,4,I like the 2nd approach because it is solution oriented and comforting as well 2nd chatbot helped me build confidence in myself and made me understand that i cant worry about factors which are not under my control that makes me feel comfortable and confident for the challenges | |
| Participant_6,rag -> non-rag,No,,10,3,4,4,4,4,"Builds relevant responses, with an approachable tone\n+ Stays on point, and addresses questions\n- Response time is very slow, takes up to 5-10sec to get answers for more complex questions\n- Tone can be overly helpful at times",14,3,2,3,4,3,"Much faster response times\n- Very brief at first, had to be told to be more helpful\n- Limited helpful questions or directions to steer the conversation\n- Tone is slightly unfriendly and could come across as aloof" | |
| Participant_7,rag -> non-rag,No,,13,5,5,4,5,5,"Clarified that it was a mental health/wellness related bot and nudged me to go back to talking about what we were discussing (wellness related) when I randomly asked a very different question in between, which is interesting. But it is able to adapt when the same prompt about the random topic is structured differently in that wellness context which is good.",15,5,4,3,5,4,"It was able to switch between different topics very smoothly. Readability isn't that good, would be nice if it used bullet points for organizing instead of chunks of text." | |
| Participant_8,nonrag -> rag,Yes,Chatgpt,9,5,4,5,4,4,,5,4,5,3,4,3, | |
| Participant_9 ,rag -> nonrag,Yes,ChatGPT a few times,6,2,3,5,4,3,I didn't find it too humanely empathetic. It just felt like a directional archive than assistant.,5,4,4,5,4,4,I found this bot to be more empathetic and focused on what I'm saying and me rather than facts and trivia. I felt more listened to with it. | |
| Participant_10 ,rag -> nonrag,Yes,"Chatgpt, I use it time to time to talk out about certain thoughts that bothers me.",7,4,4,4,4,4,I feel the experience was really great. The assistant was able to rephrase and re-iterate some of my responsed kinda validating it. It felt good as it meant that I am being listened. I also liked the way it had questions in the end to continue the conversation or engage me more on the interaction. Though some of the list of advices were unsolicited but they were good enough basic advice. It felt like it tried to problem solve in areas where I don't want advice.,6,4,3,3,2,3,"I felt talking to this assistant it was a little less humane. Like the responses were good in mirroring my thoughts or response, but the suggestion it gave was not listed as bullet points so it was hard to read. Also the assistant introduced some new words like anxiety that I never mentioned and than started giving examples like most people would do this.. kinda leading the user. This can lead to overthinking. Also it did not ask follow up questions to engage the conversation. I found the previous model better." |