"When researchers administer a sorting task, their objective is to study the natural cognitive processing of information that individuals encounter in their lives. As such, researchers must carefully considerthe specific aspects ofthe sorting taskto be used, because the various decisions may have a significant impact on the task's outcomes (i.e., the sorts), thereby influencing the conclusions that can be made about the cognitive processes they are studying."
Researchers have a lot of degrees of freedom when designing sorting tasks. How many participants does one need? How many objects should I have participants sort? Should I use pictures, or only text? Do the instructions really matter?
These seemingly small decisions may have significant impact on how participants sort, and thus create strong influence on the results you obtain. In our research, we created 36 different sorting task configurations to see the effect of such innocuous decisions. Here's what we found:
1. Researchers should keep the number of objects manageable, but there is no need to severely constrain the number of objects.
Whereas there is no doubt that participants prefer tasks with fewer objects (say 20), we find no evidence that participants cannot properly follow instructions that require them to sort 40 or even 60 objects. As the number of objects increases from 20 to 40, the biggest impact seems to be on completion time as the task is perceived to be more effortful and somewhat less enjoyable. However, the jump from 40 to 60 objects seemed to have little effect beyond making the participants feel like there is little variety in the options to be sorted. It did however decrease the rate at which participants sort the cards, suggesting that there is an increase in complexity that arises when the number of options increases to 60.
2. Researchers should be mindful of what they are asking participants to sort.
Participants are willing to engage in sorting tasks with a large number of objects. Yet they are significantly less satisfied with the task if they do not enjoy the topic at hand because the objects are perceived to be too similar, or just not interesting. We found that using groups instead of food objects led to an increase of 14% in dropout rates, nearly two minutes longer spent sorting (an equivalent number of objects), and that participants were sufficiently depleted from the sorting task that they were significantly less likely to work hard on subsequent tasks.
3. There's no harm in allowing participants to sort objects into multiple piles.
There's no harm in allowing participants to sort objects into multiple piles. Our results suggest that, when given the option, participants were likely to assign objects to multiple piles only when their perceptions dictate it. We did find that it increases the amount of time participants spend on the sort slightly and it was perceived as more effortful, but it did not affect their perception of task difficulty or their satisfaction with the task. Further, it avoids having to eliminate participants who do sort objects into multiple piles because it naturally fits their perception of the objects.
4. If researchers want labels for the piles, they should ask participants to do so after they have submitted their sorts as complete.
Our data suggests that asking participants to name the piles during the task leads to an increase in dropout rates of an average of 5%, a greater number of cards left unused by participants, and significantly longer time spent sorting.
5. If researchers are trying to increase the frequency at which participants assign objects to multiple piles, they should consider providing pictures along with the objects' names.
Doing so allows participants to visualize the objects, and tends to lead to a greater number of objects used more than once in the sorts.
6. Requiring participants to use all the cards has little effect on task and satisfaction measures.
If you suspect that participants will be familiar with the majority of the objects to be sorted, then requiring participants to sort all objects is more likely to result in fully complete data. We found that doing so had nearly no effect on the process or on how participants experienced the task.
7. Sorting interfaces are sufficiently intuitive. Expansive instructions may not be necessary.
We found that participants did follow the instructions, and that providing participants with videos that illustrated the features of the sorting interface (e.g., adding/removing objects from piles, removing them from the desk, labeling) did not have much of an impact other than to speed up the sorting process once participants got started. As a researcher, keep your instructions simple, explain clearly to participants how you want them to sort the objects, and they will likely follow your instructions.