Card sorting, also known as pile sorting, is a knowledge elicitation technique that involves participants sorting items according to some given criteria. This is done to determine how participants categorize items in a given domain and to identify similarities and differences between groups of participants. Card sorting is typically used as a research method in user experience design, information architecture, and cognitive psychology.
In user experience design card sorting is used to test a group of subject experts or users to generate a dendrogram (category tree) or folksonomy. It is a useful approach for designing information architecture, workflows, menu structure, or web site navigation paths.
The person conducting the test (usability analyst, user experience designer, etc.) first identifies key concepts and writes them on index cards or Post-it notes. Test subjects, individually or sometimes as a group, then arrange the cards to represent how they see the structure and relationships of the information.
Groups can be organized as collaborative groups (focus groups) or as repeated individual sorts. The literature discusses appropriate numbers of users needed to produce trustworthy results.
A card sort is commonly undertaken when designing a navigation structure for an environment that offers a variety of content and functions, such as a web site. In that context, the items to organize are those significant in the environment. The way the items are organized should make sense to the target audience and cannot be determined from first principles.[citation needed]
The field of information architecture is founded on the study of the structure of information. If an accepted and standardized taxonomy exists for a subject, it would be natural to apply that taxonomy to organize both the information in the environment, and any navigation to particular subjects or functions.[citation needed] Card sorting is useful when:
Card sorting is an established technique with an emerging literature.[page needed]
To perform a card sort:
In an open card sort, participants create their own names for the categories. This helps reveal not only how they mentally classify the cards, but also what terms they use for the categories. Open sorting is generative; it is typically used to discover patterns in how participants classify, which in turn helps generate ideas for organizing information. Participants in open pile sorts often differ considerably in how many groups they make. Borrowing terms from biological taxonomy (see Lumpers and splitters), those who make a few broad piles are sometimes called "lumpers" and those who make many narrow ones "splitters". This results in what is known as the lumper–splitter problem (see below).
In a closed card sort, participants are provided with a predetermined set of category names. They then assign the index cards to these fixed categories. This helps reveal the degree to which the participants agree on which cards belong under each category. Closed sorting is evaluative; it is typically used to judge whether a given set of category names provides an effective way to organize a given collection of content.
In a reverse card sort (more popularly called tree testing), an existing structure of categories and sub-categories is tested. Users are given tasks and are asked to complete them navigating a collection of cards. Each card contains the names of subcategories related to a category, and the user should find the card most relevant to the given task starting from the main card with the top-level categories. This ensures that the structure is evaluated in isolation, nullifying the effects of navigational aids, visual design, and other factors. Reverse card sorting is evaluative—it judges whether a predetermined hierarchy provides a good way to find information.
Created by Celeste Paul, The Modified-Delphi card sort is based on the Delphi method. Rather than each participant creating their own card sort, only the first participant does a full card sort of organizing and arranging items. The next participant iterates on the first participant's model, then the third participant iterates on the second's model, and so on. The idea is that with each iteration the card sort gets more refined with fewer participants and consensus is built sooner.
The purpose of card sorting analysis is to extract patterns from the population of test subjects, so that a common set of categories and relationships emerges. This common set is then incorporated into the design of the environment, either for navigation or for other purposes. There is some indication that different evaluation methods for card sorting provide different results.
Broadly, there are two main methods to analyze card sorts: semantic, where the responses of participants (card labels, group descriptions) are analyzed and syntactic, where the structure of card sorts and the placement of cards is analyzed.
Semantic methods for analysing card sorts include gist analysis and superordinate analysis. Gist analysis involves identifying group labels given by different participants that refer to the same concept; for example "Difficult to understand" and "Confusing". Superordinate analysis involves identifying the overall rule(s) used for classification by a participant for an entire card sort; for example, "Clarity of information". Superordinate analysis is sometimes referred to as identifying content analysis groups or facets.
Methods for syntactic analysis can be either structural where the overall "shape" of card sorts is analyzed (for example, the number of groups in each sort, the number of cards in each group), or item based where the relationships between cards are examined.
Similarities and dissimilarities between cards are often measured with similarity or dissimilarity matrices. Similarity and dissimilarity matrices are distance matrices that count the number of times any two cards were (in the case of similarity matrices) or were not (in the case of dissimilarity matrices) placed together in the same groups. Dissimilarity matrices can be used in cluster analysis, most often hierarchical clustering to create dendrograms.
Dissimilarity matrices can also be used to create -dimensional scatter plots using multidimensional scaling as an alternative visualisation to dendrograms.
Edit distance is a metric used to compare card sorts. The edit distance between two card sorts is defined as the minimum number of card moves needed to transform one card sort into the other. The edit distance between two card sorts can be calculated efficiently using the Hungarian algorithm.
Orthogonality is a measure derived from edit distance that is used to gauge how distinct the card sorts in a set of card sorts are from one another. Card sort orthogonality is defined as the sum of edge weights of the minimum spanning tree of the complete graph of pairwise edit distances between card sorts divided by the number of card sorts.
Edit distances can be used to identify -neighbourhoods and -cliques. These are simple clustering methods used to find card sorts that are similar to a probe sort—a card sort that is specifically constructed to examine a criterion of interest. -neighbourhoods are defined as the card sorts within a distance of from a probe sort. -cliques are defined as the card sorts within a distance of from a probe sort where no two sorts in the clique can have a distance of more than from one another.
Similarity matrices have also been used to compute distances between card sorts. Using the Jaccard distance, the proportion of card pairings that are different between two card sorts relative to the total number of possible card parings can be calculated.
The Lumper–Splitter Problem
Differences in the number of piles participants create in open cards sorts affects most of these quantitative methods. This is because a participant who makes large piles places many more card pairs together than one who makes small piles. In a similarity matrix, each participant adds one co-occurrence for every pair placed together, so lumpers contribute many more co-occurrences than splitters. When individual sorts are compared, differences in the number of piles count as disagreement even when one sort only subdivides another. For example, if one participant places six food cards (carrot, beetroot, spinach, lettuce, lentils and peas) in a single pile called "vegetables" and another divides the same cards into three groups of two called "root vegetables", "leafy vegetables" and "legumes", the first sort contains 6 × (6 − 1)/2 = 15 card pairs and the second 3 × (2 × (2 − 1)/2) = 3. All 3 of the second sort's pairs are also among the first sort's 15, which counts towards agreement between them. However, the remaining 12 of the first sort's pairs are split up in the second sort (for example, carrot and spinach are put in different piles) and thus count as disagreement.
A number of web-based tools are available to perform card sorting. The perceived advantage of web-based card sorting is that it reaches a larger group of participants at a lower cost. The software can also help analyze the sort results. A perceived disadvantage of a remote card sort is the lack of personal interaction between card sort participants and the card sort administrator, which may produce valuable insights.