Promising difficulties

At the recent VisWeek conference, Jessica Hullman and her coauthors presented “Benefitting Infovis with Visual Difficulties (pdf)”, a paper that suggests that the charts which are read almost effortlessly are not necessarily the ones that readers understand or remember best. To answer that claim, Stephen Few wrote a rather harsh critique of this paper (pdf). As I read this I felt the original paper was not always fairly represented, but more importantly, that the views develop by both parties are not at all inreconcilable. Let me explain.

What is cognitive efficiency, or “say it with bar charts”

For quite some time, we were told that to better communicate with data, we had to make visuals as clear as possible.

The more complicated way of saying that is talking of “cognitive efficiency”. By reducing the number of tasks needed to understand a chart and simplifying them, which is sometimes called reducing the “cognitive cost” or “cognitive load”, we improve all virtues of the chart.

Various charts based on the same data points, shown in order of cognitive cost

Various charts based on the same data points. From left to right, they make the task of comparing individuals value increasingly easier

For instance: bar charts are easier to process than pie charts, because it’s easier for the human eye to compare lengths than angles. So, with equivalent data, bar charts have a lower cognitive cost than pie charts. Likewise, bar charts which are ordered by value (smallest bars to largest bars) are easier to read than unordered ones. Ordered bar charts have an even lower cognitive cost than unordered ones.

Conversely, adding non-data elements add extra tasks for the reader and increase cognitive cost. These non-data elements have been reviled by Edward Tufte as “chartjunk”. His data-ink theory says that out of all the ink used for the chart, as much as possible should be devoted to data elements. Again, this goes in the direction of data efficiency.

Engagement rather than immediacy?

Again for quite some times those rules were held to be universal. Yet, several tried to challenge them, the latest being Jessica Hullman in her paper “Benefitting Infovis with Visual Difficulties“. This paper was so thought-provoking that it received an honorable mention at the recent IEEE Information Visualization Conference 2011 (as a note to the non-academic reader, this is quite a competitive achievement).

New information visualisation techniques are often evaluated.  This paper argues that such evaluations typically consider response time or accuracy, and not how well users are able to interpret and remember visuals. When only the former criteria are taken into account then cognitive efficiency is the superior framework. But this is not the case of data storytelling (which is, arguably, a small subset of  all data visualizations).

When visualizations attempt to transmit a message, then how well users can receive this message, as well their capacity to remember this for a long time are of utmost importance, much more than the ease with which a visualization is read.

In that case, Jessica Hullman proposes a trade-off between cognitive efficiency and “obstructions”. The idea is that such obstructions, or visual difficulties, can trigger active learning processes. In other words, if when trying to read a chart, a user doesn’t understand it effortlessly, but is somehow willing to get to the bottom of it, she will apply all her active brainpower to it. This effort surge will lead her to not only better interpret it but also to better remember it. To sum up, these obstructions can have positive effects, this is why when this effect works, they are called desirable difficulties.

Desirable difficulties are tricky, because if the “obstruction” is too large, if a small additional effort is not enough to understand the chart, then it will not work. So, this is definitely not about maximizing the difficulty to understand the visualizations.

In the recommendations parts of the paper the authors say:

Instead of minimizing the steps required to process visualization, induce constructive, self-directed, cognitive activity on the part of the user.

This doesn’t mean that anything goes. This paper does not argue to add as many difficulties as possible, to use every gratuitous effect in the book. Instead, the paper goes on to give actionable design suggestions to enhance reader stimulation and active information processing.

In my practice, for instance with the Better Life Index, I verify the analyses of the Hullman paper: the novelty of the form and the aesthetic appeal of the representation drive the users to overcome the difficulty posed by the unusual shape of the flower/glyph. Would bar charts have conveyed the data more efficiently and more accurately? Definitely! would the user engagement have been comparable? Definitely not.

A critique by Stephen Few

Stephen Few, whose work I have praised at multiple occasions in this blog, has published a critique of this paper (pdf). Reading his article, then the paper again, I had the feeling that they didn’t talk about the same things. In certain contexts, difficulties are not desirable at all and must be eradicated. Yet, in other contexts, cognitive efficiency does not provide the  optimal solution.

For instance, Stephen writes:

Long-term recall is rarely the purpose of information visualization.

Fair enough! so let’s agree that when it is not the case, we should not trouble ourselves with seeking to add obstructions to the display. For instance: business intelligence systems, dashboards (for monitoring), visual analytics (and more on this shortly). Spreadsheets, mostly. All usages of data that support decision, and most usages in the corporate world. The Hullman paper only applies in the other cases anyway.

He would also write (emphasis by me):

Skilled data analysts learn to view data from many perspectives to prevent knee-jerk conclusions based on inappropriate heuristics.

Agreed! and by all means, let them analyse and let them view data from as many perspectives as they see fit, and don’t get in the way of their job.

For context, check out www.palantirtech.com/government/analysis-blog/mortgage-fraud

This here is taken from a demo from Palantir government. Here analysts are tracking mortgage fraud. Each yellow dot on the top display is a transaction where a house has been sold for over 200% of its purchase value, and the ones which are connected are about the same house. We can immediately see 2 suspicious clusters where a property has been resold 4 times in these conditions. And if at the end of their work day the analysts don’t remember the address of the fraudulent transaction, it’s no big deal as long as they have identified a wrong practice.

Conversely, at the risk of repetition, the paper authors write of a trade-off between efficiency and obstructions – cognitive efficiency being generally positive. They say that obstructions become desirable difficulties only if they are constructive, that is if they are able to trigger active information processing. They are not championning 3D pie charts or atrocious dashboards as the one at the end of Stephen’s article.  Jessica signals that novelty enhance active information processing. I don’t know how to characterize speed dials in dashboards, for instance, but novel would not be the word I’d use, and again they wouldn’t be favoured by the authors of the paper. So, I think it’s a bit unfair to associate the paper with the terrible, terrible visuals presented in Stephen’s article, the ones in the original paper being a little bit more defendable.

To see this chart in context see http://www.oecd.org/dataoecd/41/50/47984536.pdf

Consider this other chart (and let’s assume for the sake of discussion that its cognitive cost is low, while it could be much lower by showing fewer time series for instance). This was published in an OECD publication almost 2 years before the 2008 crisis. I would say this chart is easy to read (we see mortgage delinquency rates dropping in most countries) but difficult to interpret and to recall. Like other charts of the document, this one is an oracle of financial apocalypse, as the proportion of delinquent mortgage in the US, the only one without a downward trend, will have the consequences that we know. So if a different way of showing the same data could have made that more obvious at the cost of legibility, I think it would have been worth a shot.

Are we on common ground yet?

If not, let’s assume now that there exists visualizations where long-term recall is, indeed, the main purpose. Examples would include use in journalism, politics, advocacy, marketing… Jessica has been involved in the series of workshop Telling stories with data at VisWeek. This suggests an interesting distinction.

  • visualizations which are tools with which a user accesses or manipulates data.
  • visualizations where an author, with a specific intent, tries to frame data in a certain way to an audience. In that case, the author wants to make sure the audience receives the message as intended, and remembers it.
See where I’m going?
In the first case, we want cognitive efficiency all the way.
In the second, we are mostly concerned with getting our message across and making it stick.
So, there is no contradiction between having a set of rules for one category of visuals, and a different one for the other, especially since the criteria of success are so different. To illustrate this I note that both the article and the paper refer to Tableau, a “cognitive efficiency” company. Yet, it turns out that Tableau is also very interested in doing as well as possible in the story telling front, and that questions asked at the paper’s presentation by Tableau representatives show their interest in this research.

Where to from there?

We have proven methods to reduce the cognitive cost of a visual, and we can thank Stephen Few for making them more accessible. It’s much more difficult, though, to optimize the characteristics of a successful “data narrative”, that is interpretation and memorability. It’s an infographics jungle out there. Those of us who haven’t seen their share of undefendable visuals just haven’t searched enough, but absolutely anything goes.
We still do not have an equivalent framework for visualizations that tell stories. InfoVis started to study them (such as in the remarkable Narrative Visualization: Telling Stories With Data) and characterize them, but we don’t have a systematic, reproducible way to make sure that data narrative will work well, just as we can do the perfect dashboard.  We do know that the best examples at large do not comply with the rules of cognitive efficiency though. Fortunately, practitioners have not waited for convincing resarch and are leading the way, even though many get lost in the process. This is why we need more research on that front! I for one is looking forward new developments in this area of InfoVis.

		    		

14 thoughts on “Promising difficulties

  1. This is truly an excellent post, Jerome! A great read!

    Before reading the actual paper, I already imagined gamification of visualiations would be a great way to add desirable difficulties, so it was great to see that ‘challenge and gameplay’ seems to be part of this paper.

    Thanks for sharing!

  2. Good argument, but I’m still thinking about it. I find that excess cognitive load may have a couple undesired consequences. First, I find I get bogged down in the details of the chart, and more its construction than its contents. Second, if something becomes too obscure, I may abandon entirely my attempts to interpret it.

    The chat used to detect mortgage fraud does not have what I would consider excess cognitive load. I don’t know the details, but I can’t imagine they are obscured by the presentation.

    Are you trying to convey information in those flowery glyphs? At first I was distracted by the lobes, since they remind me of electron orbitals. Then the thought that the petals are encoding information frustrates me, because I’ve yet to see this approach (glyphs) have any effectiveness. So this chart encounters both of the reactions I described in the first paragraph.

  3. Jerome, I am worried I am getting to much competition from your blog! ;-) Your posts are getting really excellent, thank for your great job, keep the good work.

    I’d love to talk more about this paper and I don’t have time now to write a long essay. I think there are additional aspects to consider, it’s not that easy.

    In my opinion the problem with the paper is the idea of “difficulty”. I don’t buy it that we need to artificially add difficulties. But the trick is that what they call difficulties might not be a difficulty at all. If I have to think about excellent visualization designs with a non-orthodox cognitive-efficiency style, I won’t say they add visual difficulties. I don’t think efficiency is to be traded off with aesthetics or beauty or similar elements.

    On the other hand I do criticize the use of cognitive-efficiency as the only factor to judge a visualization but for other reasons. One above all is that the difference between a solution or another in terms of efficiency is totally negligible and the one that is a little bit more effective might be preferable for other reasons.

    Also, complex data analysis systems are just not prone to be evaluated under this lens solely just because the reasoning process, so long, and so intertwined with other elements of the analytical environment that it’s silly to believe they are the most relevant factor.

    Anyway … again great posts!

  4. Nice post. It is obvious that data visualization is a language with multiple dialects and in some of them effectiveness may not play a central role. I like the concept of “desirable difficulties” but that’s a Pandora’s Box that not many are able to open.

    Google for “memorable charts” or “professional-looking charts” and you’ll get a lot of undesirable difficulties. Your “business charts” dialect may emphasize effectiveness but if your “role model” is a sexy infographic you’ll try to copy it for the next business presentation, with disastrous results.

    The “effectiveness dialect” (Cleveland’s experiments + Tufte’s minimalism) is relatively stable and consistent. We can’t say the same about the “emotional dialect” (for lack of better word) where you can find good examples like the Better Life Index but also many bad examples like a Dundas pie chart. Graphic designers must find a way to draw a line between them. “Level of engagement” is in the equation, but the average user needs something more.

  5. Hi Jorge, I think we can all agree on “undesirable difficulties”. Social networks make these charts more visible now and I sometimes feel like Pandora’s box has been opened. There is a form of unhealthy emulation and escalation in outrage to common sense.

    The reason why I wrote this is not to defend such practices, although to be honest I’m quite grateful that designers out there try such extreme forms because they can be studied. But rather, to say that desirable difficulties is something entirely different. They are speed bumps, rather than roadblocks. It’s not about trying to deviate from commonly accepted representations as much as possible. And in fact, we don’t quite know what’s the best way to set them up. that’s why I am looking forward to more papers on this!

  6. well, this subject has been on our common agenda under different guises. so I suppose we will discuss it live next time we meet!

    you are raising good questions and I struggle to give a concise answer. I agree that the notion of difficulty is counter-intuitive, yet I’ve seen this pattern outside of visualization so I can accept it in theory, with the caveat that not just any difficulty can work. I note however that not everyone agrees on what constitues cognitive cost. For instance, Few and Hullman disagree about small multiples vs animation. Hullman says that small multiples have a higher cost because passively viewing a flow of images requires less effort than acquiring information piece by piece by looking at each small multiple in turn, in other words receiving information is easier with animation. But Few says that small multiple have a lower cost because comparing across several multiples – which are there to see – is much easier than across several frames of animation, so manipulating the information is easier with small multiples.
    My intuition is that visual difficulties may work very early in the process (ie a non-orthodox, yet grammar-of-graphics complient representation) rather than later (no need to annoy the user once they are sold in the concept). Yet again I see NYT graphics holding the hand of the viewer and effectively controlling the pace throughout the whole time.
    the great thing is, no matter what, in the next year we’ll see thought-provoking visualizations or apps that will try to implement that. Some will work, some will fail, but certainly there will be someone to comment!

  7. Absolutely – excessive cognitive load is a game-stopper. those “desirable difficulties” have to be moderate. If the slope is too steep, even if I accelerate I can’t get past (or so I’ve heard, I don’t drive. Is that how cars work?). and just because this effect has worked in some settings it is definitely not a license to try anything.
    I’ve used the mortgage fraud thing as an example where there is little cognitive load and where preattentive attributes work. Even if you don’t know what this is about you see instantly the 2 crossed squares and you get the intuition that they are points of interest. It would be a pity if the system somehow got in the way of that and added chartjunk or whatever so you could interpret this better. A Palantir analyst who is able to create this view is not likely to appreciate the attention.
    for the flower things, you’re right – the main point is not really to convey information, or at least not in the petals. But the goal of this specific visualization is the means in desirable difficulties, that is to create 5 minutes on that page on average, versus a handful of seconds elsewhere on pretty much any page of the OECD site. Through this visualization I was comforted in the idea that different representations can engagement. It would not have been too difficult to represent every data point used to create the flowers in a very neat dashboard. Would we have had the same level of user commitment? People stay aroundreally enhance engagement which can in turn have benefits, although, yes, it is to the expense of the transparency of the information.

  8. Hi Jan Willem, this reminds me that I should post something on my own intervention at visweek which was precisely on what you say, hopefully sometimes before visweek 2012
    cheers!

  9. Jerome, a few years ago I wrote a post in my Portuguese blog titled “Can a good chart be dangerous?” (“good” in the sense of “effective”). To tell you the truth, actively adding a speed bump to reduce a chart effectiveness never crossed my mind, but I wrote that an overly optimized chart could trigger some adverse side effects (passive acceptance).

    As you know, from the “effectiveness” point of view adding desirable difficulties amounts to heresy. Solving the problem with data (adding more detail) is a more palatable option, but it’s the end result that matters, and we shouldn’t close the door to lateral thinking. We really need it.

  10. Hi Jerome, good article. I think you’re spot on with the distinction of visualizations.

    But I propose new metaphors:
    – Instead of “speedbumps”. Desirable difficulties are “Narrow roads”. (Speedbumps remind us of the uncomfortable feeling of bah-bump as the car stumbles over a that lump in the road.)
    – Cognitive efficiency will then be a “highway”.

    Cheers!

  11. Hi Thorri, well the idea is that it makes you slow down but after you accelerate. maybe a dog crossing the road? hopefully those are not associated with bah-bumps in the collective mind :)

  12. Great post! Excuse me, if the following stream of thoughts lacks context…

    Having to reconcile conflicting forces is actually one of the key aspects of cognitive science. Heck, a whole theory to explain language acquisition and communication that is based on a conflict resolution heuristic has become the mainstream theory of almost all linguistic disciplines. It would be really surprising if visualization were exempt from this phenomenon.

    Much like OT poses for utterances I would assume that in visualizations there’s a sweet spot, a goldilocks position of difficulty, that works best for the specific constraints of a given task. Since a visualization is but a very elaborate signifier for a complex proposition the cognitive mechanisms for communication apply which means, that the degree of difficulty to parse the signifier will be interpreted by the recipient to be indicative of something.

    When we look beyond what is overtly stated by the semiotic entities of a visualization we are entering a meta level of communication. We enrich the meaning of signifiers beyond their truth values by contextualizing world knowledge and our expectations of the communication game – the assumption that the sender of a message actually wants to engage in meaningful conversation being just one of them.

    So without having a clear idea of how to apply the cross-disciplinary theorems to visualization I’d like to just throw one more thing out there, regarding the paper in question. If we look at the concept of cognitive dissonance, we know that it has been utilized to make messages captivating and memorable. Perhaps creating this feeling of discomfort in the mind, nudging it to reiterate its assessment of the signifiers put in front of it, is what the right degree of “difficulty” accomplishes.

  13. Excellent piece, Jerome. Thank you for writing it, and thanks for your careful scrutiny of the subject.

    Do you think a part of this discussion relates to the expectations of the nomenclature? It seems that the phrases “information visualization” and “data visualization” are getting used interchangeably, when they probably shouldn’t. When Stephen Few says “Long-term recall is rarely the purpose of information visualization”, do you think he really means “Long-term recall is rarely the purpose of data visualization” ?

    Data seems to lend itself (in general) to short-term recall (dashboards, instrument readings, etc) while information is something that might take longer to process, and be a benefit a person more in the long-term.

    Things get complicated when visualizations provide both data and information, which many do these days, and still more complicated when people try to discuss them as a single thing, with a single purpose.

    The work we do at Periscopic never really starts with a discussion about which type of visualization we will make, but rather what kind of data and information will best inform the subject. Not sure if that is a good or a bad thing… must think more about it.

    It’s exciting to see these concepts evolve and be refined publicly like this. Thanks again for the post.

  14. Nice article Jerome. One of the messages or examples we have been using for the last year when talking about various styles of visualisation and when focused on story telling or exploratory visualisations is that of Angry Birds. Angry Birds is not a straight forward game in the later levels, with multiple bird types, different actions, the need to get angles right etc. But the first levels are…the game achieves something very few data visualisations do which is immediate engagement ultimately leading to education of the user/player/viewer ultimately giving the viewer the skills and knowledge necessary to handle more complex levels. This is common in many good games and increasingly so in casual games but is at its peak in Angry Birds.

    While I won’t dismiss the need for “academic papers” on this topic, I think that if we looked a little outside the realm many visualisation academics focus on and sought to learn from what the arena of Game Design we would potentially jump start a lot of this understanding and ultimately create visualisations which are at the same time immediately engaging with a rich narrative for exploration underpinning it.

Leave a Reply