Usability testing kit
A collection of guidelines and templates to help you check the usability of election materials
Usability testing is a way to learn how easy or difficult it is for people to use something by observing them actually using it.
The resources provided in this kit can help you systematize the process, produce reliable findings, and generally expand your ability to test voting materials thoroughly with real users — before you launch them out into the world.
What you'll need
This kit covers 3 different approaches to usability testing. Different approaches require slightly different materials, but these are the basics that will get you started:
- Someone to run the test
- Someone to help observe and take note
- Someone who might use what you’re testing
- Something you want to test (a newly designed ballot, a revamped election website, new voting equipment, etc.)
- A quiet place to conduct the test
Table of contents
Getting familiar with usability testing
In a usability test, you show materials to individuals, one at a time. Asking each person to use what you’ve given them, you watch and listen without teaching or training.
That’s different from focus groups, surveys, and advisory groups, where you might show the materials to groups of people and ask them what they think or how they feel.
Why do usability testing
Through usability testing, you can learn about what works (and what doesn’t) for users, and you can confirm or challenge assumptions.
With data from usability testing, you can update your election materials, design a ballot, or launch a new outreach campaign with confidence.
It will help you:
- Identify problems in the design of election materials
- Make it more likely that people will have efficient, effective, and pleasant interactions with your office
- Make it easier for users to use your materials – whether the users are candidates, voters, poll workers, election staff, advocates, media, or other people your office interacts with
When to do usability testing
You can do usability testing almost any time, but it’s most useful:
- Before you start redesigning something
- While you’re still working on a draft
- When a significant event happens that may cause changes or require staff or volunteers to be retrained
- When something about the voting situation has changed since the last election
Planning your usability test
What does a usability test look like?
A usability test is made up of one-on-one sessions with one moderator and one user or participant. The moderator (usually you) will be near the participant to conduct the session and ask follow-up questions (but not to teach or help). A note taker may support the moderator by logging observations and discussion.
During the session, the moderator and note taker collect observations (what you see and hear). Observations give you data that you can use to make judgments about what’s frustrating to users and how you might eliminate those frustrations.
Usability session with moderator, note taker, and 3 participants
Each participant uses a copy of the same materials. The moderator guides participants through the session, observes their behavior, and takes notes (if possible, which is why you have a note taker). Remember: do not help participants do the tasks.
After the testing part is done, if there’s time at the end of the session, you may help participants learn how to use the materials. They often appreciate knowing what their mistakes were. Correcting participants will help you understand what to fix in the materials, too.
Usability test sessions are usually short – about 15 or 20 minutes – but can go longer.
Consider inviting other people on your team to join in by observing the sessions (but silently).
How many participants do I need?
Some rules of thumb:
- Try to test at least 4 people who normally might use the material you’re testing. If you think there might be different behaviors for different types of users, include 4 of each type.
- Make sure your participants represent a diverse group of users. This means including people with disabilities, different ethnic backgrounds, and differing English language proficiencies.
- Don’t worry about demographics like age, education, or income. Think more about what experience participants might have doing the tasks you want to observe, and simply recruit a range.
- You don’t need a large sample of participants. Don’t worry about “statistical significance.”
The goal is to learn enough in your usability test to make informed design decisions.
How should I find participants?
First, you have to decide if you’re going to schedule people in advance, or if you’re going to use impromptu recruiting by going to where likely participants might be.
If you’re scheduling appointments ahead of time, tell everyone. Use every channel you’ve got to invite participants.
Usability testing at a community center
If you’re doing quick testing in the field, go to places like community centers, county offices, shopping areas, farmers markets, and libraries. You want to be in a place with a lot of foot traffic.
No matter what approach you take to recruit your participants, make sure they fit your criteria of likely users. That means that they currently do the behavior you want to observe, such as voting, running a campaign, working the polls, and so on.
What should I look for during a usability test?
You’re looking for both what worked (successes) and what didn’t work (failures). For example:
- Any errors or mistakes – even if the participant corrects them
- Failure to complete or submit
- Places where the participant didn’t follow instructions
- Any signs that the participant doesn’t understand the information or actions needed
- Any errors specific to the material being tested
Behaviors you might see and want to take note of:
- Questions participants ask
- Requests for assistance
- Spoken comments or non-verbal reactions
- Expressions of emotion (confusion, frustration, anger, disgust, delight, satisfaction)
- Any behavior that is adaptive (such as taking out reading glasses or moving in closer to the machine)
- Incidents that would be documented and recorded
- Behaviors or comments that surprise you
- Things that surprise the participants
- Distractions (glare, noise, movement)
Holding a practice test
Before testing with real volunteers, consider a practice session with colleagues or friends. This exercise will also help you practice note taking.
Since you might not already have materials you want to test, we’re providing a sample: instructions for how to fold an origami rooster.
Your goal with the test? To measure the effectiveness of the origami instructions provided.
What you’ll need
- Moderator
- 3 participants
- Paper sheets to make an origami
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials.
Download the practice test materials (.zip file, 426KB)
Duration of test
Not more than 15-20 minutes per participant
How to do the test
Find a quiet spot to conduct 3 usability tests — one participant at a time. Share the instruction sheet with them along with the paper supplies to make the origami.
Practice test instructions
Follow the moderator’s guide provided to introduce the participant to the project and guide them through the process. While they’re testing, jot down observations.
After you’ve completed all 3 sessions of your usability test, use the note taking guide provided to record your top 5 observations and insights about the usability of the origami instructions, focusing specifically on what you heard and what you saw. In the “additional comments” section, include points of analysis on the closing questions:
- What worked well?
- What needs improvement?
Once you’ve practiced with this test session, you’ll be ready to look at the 3 approaches to usability testing that we cover here and think about which one best suits your situation and your needs.
Table of contents
- Overview: 3 approaches to usability testing
- Testing in the office: quick usability feedback from colleagues
- Testing out of the office: catch real users in the field
- Testing in the conference room: thorough, structured testing with real users
Overview: 3 approaches to usability testing
There are lots of ways to conduct a usability test, and the approach you take will depend on what you want to learn and the amount of time and resources you’ve got.
We’re highlighting 3 approaches, ranging from quick and easy to thorough and structured:
- Testing in the office: quick usability feedback from colleagues. If you’re in the early stages of designing your materials, this approach is a fast way to get a second opinion.
- Testing out of the office: catch real users in the field. If you have drafts or prototypes that are close to complete, use this approach to test how people use them.
- Testing in the conference room: thorough, structured testing with real users. If you have solid, revised materials, this approach allows you to run a structured test with real users who you’ve scheduled ahead of time.
Below, we provide details, guidelines, and materials for each the 3 approaches below
Testing in the office: Quick usability feedback from colleagues
Use this approach any time you want quick feedback on something you’re developing. Of the 3 approaches, it requires the least time and resources and usually works best in the early stages of designing new materials or updating old ones. It’s also good exercise if you’re new to usability testing.
Informal usability testing in the office
Who to test with
Bring together 2-4 colleagues – but make sure they’re people who know nothing about the material you’re testing. Although your colleagues need not represent your target audience, you can ask them to use personas to think about a user’s perspective.
What you’ll need
- A moderator
- A rough draft (even a sketch) of the new material you’re developing
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials.
Download the Testing in the Office materials (.zip file, 41KB)
Where to do it
Since it’s informal, you can do this test at your desk, a participant’s desk, or in a conference room.
How to do it
- Recruiting – Walk down the hall or to the next department to find someone who’s not familiar with the material. Ask the person if they have 15 minutes to spare to do a quick review of the material with you.
- Setting up – If they agree, find a quiet spot to conduct the test. Take out your moderator’s guide as well as the materials to be tested. Using just a notepad or the note taking guide provided, jot down observations and comments.
- Testing – Hand them the sketch or draft design and ask them to use it as a typical user might. Pay attention as they work their way through the material, jotting down any usability issues that come to the surface during the testing.
- After the test – Thank them for their time.
- Reporting – There’s no need to formalize your findings; just use your rough notes on usability issues to revise the materials or start developing a prototype.
Testing out of the office: catch real users in the field
Use this approach if you’re looking for quick input on prototypes that people can try out. This test is more structured than testing in the office, but it’s quicker and easier than the testing in the conference room approach.
Who to test with
Your participants should come from the public, so they’re more like your real users (than your colleagues usually are). For example, if you’re testing something for voters, you can ask people if they’re registered to vote and go from there.
Depending on the time you have available for each session, you can work with anywhere between 4 and 8 participants for this test. You don’t need to pay your participants, but you might consider giving a small token of appreciation — like a promotional item provided by your election office.
What you’ll need
- A moderator
- A prototype/early version of the new material you’re developing
- Name badges
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials.
Download the Testing Out of the Office test materials (.zip file, 381KB)
Where to do it
The idea for this test is to leave your office and go to a public place. That might be the foyer in your municipal or county building, a public library, or a community center.
If you have some flexibility with location, find a place where you’re likely to find people who are like your real users.
Usability testing “in the field” at a community event
How to do it
- Setting up – Set up a table at the location. Be sure to use signs and badges to tell people what you’re doing (“Help us improve our election materials!”).
- Recruiting – Wearing your badge and a smile, walk up to someone and ask if they could give you a few minutes to help voters in the community. Ideally, you want to find participants who would typically use the materials you’re testing.
- Testing – When they’ve agreed to help you out, have each participant complete the consent form and demographic questionnaire. Then, begin the usability test. Pay attention as each participant works their way through the material. Observe their behavior and make note of their comments as they go. When they’re finished, ask them to tell you 2 good things about the materials and 2 things that could be improved.
- After the test – Review your notes and make sure you haven’t missed anything you wanted to observe or follow up on. Thank your participant for their time and give them a copy of the “thank you” card.
- Reporting – Depending on your situation, you can use your findings to compile a short report or simply use your notes to iterate your design further.
Testing in the conference room: thorough, structured testing with real users
Use this approach if you want to conduct a more structured test with participants who are your actual users — not just colleagues or people on the street. A full usability test can happen at any stage in the process, from testing the original version, testing drafts, or testing the almost-final version. This approach may or may not result in a formal report.
Who to test with
For this test, select participants who represent the intended audience for the material being tested. Screen them through phone conversations and schedule their sessions.
You should test with at least 4 participants per user group who will interact with your materials (e.g., 4 regular voters, 4 new voters, etc.). Full usability tests can be conducted with anywhere between 8 and 100 participants. Most of these usability tests include 8-20 participants in individual sessions that last between 30 and 90 minutes. Most will be about an hour long.
In-depth usability testing in the conference room setting
Depending on your resources, you may choose to compensate participants for their time, especially if each test is a long session of 30 minutes or more.
What you’ll need
- A moderator
- A helper/note taker
- A complete copy of the material to be tested
- Name badges
- Test materials (see below)
In some instances, additional observers may also be included in the testing room. Or, you can use a webcam to have stakeholders observe from another room.
Test materials
To run this test, you’ll need to download the test materials
Download the Testing in the Conference Room test materials (.zip file, 1.2MB)
Where to do it
Find a quiet place for testing (like a conference room) that will allow you to have substantial, uninterrupted conversations with each participant. If your test includes observers, make sure there’s room for everyone. Alternatively, you can also schedule sessions in a usability lab (two rooms divided by a glass panel to keep participants and observers separate).
How to do it
- Recruiting – Recruit your participants in advance. Schedule each session for between 30 and 45 minutes to give you enough time to conduct the test, jot down notes, and get ready for the next participant. We recommend having at least a half hour break between sessions.
- Setting up – Get to your testing location early to set up. Using the test materials, you’ll have a number of different forms to keep track of. Having them laid out prior to testing will ensure a smooth session. Make sure to use the moderator’s guide and note taking guide provided so that questions and answers can be systematically recorded.
- Testing – First, have each participant complete the consent form. You can have them fill out the demographic questionnaire before or after the test. Following your moderator’s guide, carefully observe how each participant interacts with the testing material as well as any comments/gestures they make during the process.
- After the test – Thank them for their time and give them a copy of the “thank you” card on the way out.
- Reporting – Using the report template provided, compile your results into a report, highlighting all stages of the usability test — i.e., participant data, testing locations, material tested, findings, and recommendations.
No matter which approach you take, a usability test will give you important information about how real users interact with your materials, helping you to make informed decisions about how to update or fix issues before you launch the materials out into the world.
You can think of each test as a step toward a more intuitive voting experience!