...
Our first test user is a senior, majoring in course 16.
Task 1: The user create created a new note in the 6.813 folder and titled it "Animation." He did not immediately start recording video: first he tried pressing the disabled play button. He clicked the enlarge video button, and began to take notes.
...
Aside from initial confusion with the play and record button, the user did not make any errors in using the interface. He said that the most irritating part of it was how enlarging the video while taking notes made you have to scroll sideways after inserting a large snapshot, and pointed out that the smaller video size was not useful to see what was going on while taking notes. However, since in a functional implementation the video on screen would match up with what was happening in the real world, enlarging the video while taking notes would probably not be a common case. He also was surprised that the enlargement button did not make the video go full screen.
User 2
User 3
...
Most of the usability problems he encountered had to do with our video resizing behavior. Perhaps eliminating the confusing enlarge button (which is too similar to a full-screen button), in favor of making the Record and Review pane resizeable as a whole so the users can control the size of the video would be better.
User 2
Our second user was a sophomore majoring in course 11.
Task 1: The user created a new note in the 6.813 folder and titled it "Animation". She clicked the play button when trying to record, rather than clicking the record button. She was very confused by the idea that she was supposed to pretend she was in a lecture with the ability to record a video of the lecture, because it looked to her like she was just watching a video. This probably wouldn't be an issue if the user was actually in a lecture with the ability to record the live video feed.
Task 2: The user enlarged the video to take a snapshot, likely in the hopes of getting a larger picture to insert. She kept the enlarged version of the video player for the rest of the duration, but this wouldn't really make sense in a real usage scenario because the lecturer would be visible and the video would not need to be large. She had no problems cropping the image.
Task 3: This task was easy because the user had already created the note in the correct folder. This user also had issues with the "Home" button taking her to the root rather than the parent folder.
Task 4: When trying to find the notes, she initially didn't try to sort the notes by title, but she did eventually discover the search bar. However, searching for "Lecture 19" didn't find anything, because the search only works for note contents. She then discovered the sort function, but the sort order was broken because it sorts "Lecture 1, Lecture 10, Lecture 11, ... , Lecture 2, Lecture 20, ..." Instead of increasing order. When she finally got to the note, she didn't click on the relevant section of text to skip to that part of the video, she instead dragged the scrub bar until the right section was highlighted, which took quite a while because it was hard to be accurate enough.
Overall, the user didn't have serious issues with the core functionality of the interface, it was just supplementary features that were confusing. Most of the issues with the core functionality were learnability issues, so the next time she used the application she would probably better understand how it worked. Resizing was also an issue with this user, so maybe eliminating the enlarge video button and replacing it with a resizable Review pane would be a good idea. Since the user wanted to go back to the parent folder from the note and couldn't do it with the "Home" button, she was worried that clicking the browser back button would lose all of her changes to the note.
User 3
Our final user was a junior majoring in course 6.
Task 1: The user made a new note in the root folder, hoping to be able to move it into the 6.813 folder and was surprised that it went directly to the edit screen. This user had no issue with starting the recording, noting that the disabled "play" button was helpful in letting him know what he could and couldn't do.
Task 2: While this user didn't resize the video player to take a picture, he had much more trouble actually cropping it. Part of the inserted image was hanging off the bottom of the screen, and since our "crop" and "delete" buttons always show up just below the image, they weren't visible so he didn't think there was an option to crop at first. He also got annoyed at the "crop" and "delete" toolbar moving when he scrolled, rather than remaining attached to the bottom part of the image.
Task 3: Since the user didn't create the note in the 6.813 folder to begin with, he had to move it to the correct location. He had no problem dragging it into the correct folder.
Task 4: This user also tried to use the search bar to find "Lecture 19", which didn't work because it only searches based on note contents. Once he found the note, he was unsure of how to find the relevant section of notes. He started watching the video at 2x speed for a few seconds, before realizing that the scrub bar could be dragged. Once he learned (by mistake) that the correct section of video could be skipped to by clicking on the corresponding notes, he was able to navigate the notes and video much more easily.
Again, most of the issues involved seemed to be caused by learnability problems. Once he used the application a few times, the user would likely have no problem accomplishing these tasks.
Usability Problems Summary
- Users clicked the disabled "play" button rather than the "record" button when trying to record new video.
- Category: Learnability
- Severity: Minor
- Proposed Solution:
- The "expand video" button looks like a "fullscreen" button
- Severity: Cosmetic
- Proposed Solution: Replace with a different icon that looks less like fullscreen
- Sizing issues arose when the user made the video bigger. Snapshots in the text editor became too big for the smaller editing window.
- Severity: Major (Visibility suffered greatly, and navigating the editor window became difficult)
- Proposed Solution: Rather than having an "expand video" button, allow the division between the editor pane and the playback pane to be draggable to resize the two areas.
- The "Home" button takes users back to the root of the file tree, rather than to the parent folder like they might expect. To go back to the parent folder, they might use the browser's "back" button but they worry that their changes will be lost.
- Severity: Minor
- Proposed Solution: Provide a back button within the interface (separate from the browser back button) so that the user feels safer going back one level. Since this button would be part of the application, rather than part of the browser, they might not feel like their changes would be lost by using it.
- The search bar in the file browser doesn't search by lecture title, only by note contents.
- Severity: Major
- Proposed Solution: Include the lecture titles in the searchable fields
- The sort option sorts numerical fields in the wrong way.
- Severity: Minor
- Proposed Solution: Correct the sort order for numbers.
- The "Crop" and "Delete" toolbar for cropping images always displays at the bottom of the image, sometimes off the bottom of the page. Also, this toolbar scrolls with the page rather than remaining attached to the bottom of the image.
- Severity: Minor
- Proposed Solution: Fix the positioning of this toolbar, and make it reposition correctly when the window scrolls.
Reflection
Overall, the way that we did our initial designs and paper prototypes had some pros and cons. We came up with three very different ways to bring images into our note-taking program: the webcam solution we implemented, taking photos with a smartphone, and drawing diagrams with a smartphone. We paper prototyped all three, since they were so different, before settling on the webcam option since we liked the extra review features it let us implement. Considering very different alternatives for how to handle our input was useful, but it also meant that we only came up with one design option for the input method we ended up using. Paper prototyping all three alternatives let us be a lot more confident in our decision to narrow in on the webcam alternative, but it also meant we spent less time developing and iterating our paper prototype of the webcam design. In summary, the broadness of our scope in the beginning of our design and paper prototyping process helped us come upon a design idea that we might not have decided on otherwise. However, it left us with less time to tweak and iterate on that idea once we had settled on it. The ideal solution would be to start with a broad scope, but spend more time iterating later; however, time constraints prevented us from doing this. It's difficult to say whether we made the right choice in that regard. However, even with less iteration on our paper prototype of the webcam design, we still got very good feedback from users, and made several major design decisions based on their feedback, detailed in the Design section.
Our computer prototyping focused more on the editing interface than the file browser interface, since the file browser was not very necessary for our tasks and was in our opinion a less interesting interface for us to develop, since we were keeping it fairly externally consistent with similar interfaces. However, our heuristic evaluators had a lot to say about the file browser, and if we had spent more time implementing its behavior fully for the computer prototype we could have gotten more useful feedback from them in that regard. We probably missed out on some critiques they might have made of our final file browser design. However, since the Editing interface is far more important to our project, we don't regret cutting out prototype features from the editing interface to focus on the file browser.
Overall, we're satisfied with the tradeoffs we made when constructing our paper and computer prototypes for users and heuristic evaluators. However, in an ideal world, we would have been able to devote more time to our prototypes: as it is, we're sure we missed out on some useful critiques people weren't able to give us because of incomplete prototype implementations.