...

• Are these buttons or drag and drop?
• What happens if you make a mistake? -> jumped ahead in tasks
• Successfully fixed a mistake without hesitation
• "Prototype is very thorough"
• Restarted previous race without hesitation
• "Race" -> last race or previous ballot?
• Do you need to record who the write in was?
• "Does the progress bar measure races in a ballot or across all ballots?"

Round 2

User 1:

• Where do you enter write-in? 
• Doesn't see candidate names as buttons to click on
• Restarted previous ballot opposed to just clicking race/candidate name
• Understood the second time that you can click a name and not just reset whole ballot
• Like the fix mistake button.

User 2:

• Can tell that in the main screen, right is for selection and left is for verification
• "You have to reenter all information?" when going back to fix mistake
• Expected to change just one race and return to same location
• Expected to see the panel on the right - "You work on the left and see a review on the right"

Users 3 and 4:

...

In Round 2, we focused our testing on the "Fix Mistake" menu, so our critical incidents were centered on that part of the prototype. 

Users liked the large "Fix Mistake" button and found it easy to enter that menu. After fixing one mistake, users were generally able to use the menu accurately and efficiently. 

Even though the help menu was displayed constantly when the user went to the "Fix Mistake" menu, many users did not read it carefully. There was some confusion about the reset button next to the ballot heading; some users thought they would correct an error and then hit "Reset" to save their changes. Some users clicked the "Reset" button on the whole ballot instead of correcting one individual race. Users were also surprised by the sudden input mechanism on the left. Previously, they had been using the right panel to enter in the ballots. When that panel became informational and the user was expected to click on the left panel, there was some confusion. In addition, the help menu was written vertically and some users read the items sequentially, as if they had to complete every step in order to fix their mistake. 

...

Prototype Iterations

Most users found the main screen easy and intuitive to use, but had lots of trouble understanding the "Fix Mistake" menu. As a result, we focused on changing that menu in our second iteration. The terminology of auditing is confusing, but our testers never chose the "Help" menu. As a result, we now just display the instructions on the right hand-panel to make sure all information is shown from the start. In addition, instead of lots of buttons requiring understanding of auditing terms, the edit button now shows up beside each ballot, and each race/candidate name becomes a clickable button. Now users can just click on what they need to edit without knowing the name for it. The bottom buttons of the left panel include "cancel" and "restart whole audit" to make these two items distinct. We also added a confirmation page so any choice informs the user what proceeding will do the audit, and gives them a chance to cancel. This improves the system safety.

...