Jeffrey Heer
Michael Bostock
ABSTRACT
Understanding perception is critical to effective visualization design. With its low cost and scalability, crowdsourcing presents an attractive option for evaluating the large design space of visualizations; however, it first requires validation. In this paper, we assess the viability of Amazon's Mechanical Turk as a platform for graphical perception experiments. We replicate previous studies of spatial encoding and luminance contrast and compare our results. We also conduct new experiments on rectangular area perception (as in treemaps or cartograms) and on chart size and gridline spacing. Our results demonstrate that crowdsourced perception experiments are viable and contribute new insights for visualization design. Lastly, we report cost and performance data from our experiments and distill recommendations for the design of crowdsourced studies.
- V. Beattie and M.J. Jones. The impact of graph slope on rate of change judgements in corporate reports. ABACUS, 38:177--199, 2002.Google Scholar
Cross Ref
- J. Bertin. Semiologie Graphique. Gauthier-Villars, 1967.Google Scholar
- D.M. Bruls, C. Huizing, and J.J. van Wijk. Squarified treemaps. In Data Visualization 2000, Eurographics/IEEE TVCG Visualization Symp., pages 33--42, 2000.Google Scholar
- C. Cheshire. Selective incentives and generalized information exchange. Social Psychology Quarterly, 70, 2007.Google Scholar
- W.S. Cleveland. Visualizing Data. Hobart Press, 1993. Google ScholarDigital Library
- W.S. Cleveland, P. Diaconis, and R. McGill. Variables on scatterplots look more highly correlated when the scales are increased. Science, 216:1138--1141, Jun 1982.Google ScholarCross Ref
- W.S. Cleveland and R. McGill. Graphical perception: Theory, experimentation, and application to the development of graphical methods. J. Am. Statistical Assoc., 79:531--554, 1984.Google ScholarCross Ref
- F. Cole, K. Sanik, D. DeCarlo, A. Finkelstein, T. Funkhouser, S. Rusinkiewicz, and M. Singh. How well do line drawings depict shape? In ACM SIGGRAPH, pages 1--9, 2009. Google ScholarDigital Library
- B.D. Dent. Cartography: Thematic Map Design. William C. Brown Publishing, 1998.Google Scholar
- W.R. Garner. The processing of information and structure. Erlbaum, 1974.Google Scholar
- P.P. Gilmartin. Influences of map context on circle perception. Annals Assoc. of Am. Geographers, 71:253--258, 1981.Google ScholarCross Ref
- J. Heer, N. Kong, and M. Agrawala. Sizing the horizon: the effects of chart size and layering on the graphical perception of time series visualizations. In ACM CHI, pages 1303--1312, 2009. Google ScholarDigital Library
- Panos Ipeirotis. Mechanical Turk: The Demographics, 2008. behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html.Google Scholar
- A. Kittur, E.H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In ACM CHI, pages 453--456, 2008. Google ScholarDigital Library
- R. Kohavi, R.M. Henne, and D. Sommerfield. Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In ACM KDD, pages 959--967, 2007. Google ScholarDigital Library
- R. Kraut, J. Olson, M. Banaji, A. Bruckman, J. Cohen, and M. Couper. Psychological research online: Report of board of scientific affairs' advisory group on the conduct of research on the internet. American Psychologist, 59:105--117, 2004.Google ScholarCross Ref
- S. Lewandowsky and I. Spence. Discriminating strata in scatterplots. J. Am. Statistical Assoc., 84:682--688, Sep 1989.Google ScholarCross Ref
- K. Ling, G. Beenen, P. Ludford, X. Wang, K. Chang, X. Li, D. Cosley, D. Frankowski, L. Terveen, A.M. Rashid, P. Resnick, and R. Kraut. Using social psychology to motivate contributions to online communities. Journal of CMC, 10, 2005.Google Scholar
- A.M. MacEachren. How Maps Work: Representation, Visualization, and Design. Guilford, 1995.Google Scholar
- W.E. Mackay, C. Appert, M. Beaudouin-Lafon, O. Chapuis, Y. Du, J.-D. Fekete, and Y. Guiard. Touchstone: exploratory design of experiments. In ACM CHI, pages 1425--1434, 2007. Google ScholarDigital Library
- J.D. Mackinlay. Automating the design of graphical presentations of relational information. ACM TOG, 5:110--141, 1986. Google ScholarDigital Library
- J.D. Mackinlay, P. Hanrahan, and C. Stolte. Show Me: Automatic presentation for visual analysis. IEEE TVCG, 13:1137--1144, Nov/Dec 2007. Google ScholarDigital Library
- W.A. Mason and D.J. Watts. Financial incentives and the "performance of crowds". In KDD-HCOMP, 2009. Google ScholarDigital Library
- D.F. Reilly and K.M. Inkpen. White rooms and morphing don't mix: setting and the evaluation of visualization techniques. In ACM CHI, pages 111--120, 2007. Google ScholarDigital Library
- M.J. Salganik and D.J. Watts. Web-based experiments for the study of collective social dynamics in cultural markets. Topics in Cognitive Science, 1:439--468, 2009.Google ScholarCross Ref
- B. Shneiderman. Tree visualization with tree-maps: 2-d space-filling approach. ACM TOG, 11:92--99, 1992. Google ScholarDigital Library
- B.G. Shortridge. Stimulus processing models from psychology: Can we use them in cartography? The American Cartographer, 9:155--167, 1982.Google ScholarCross Ref
- D. Simkin and R. Hastie. An information-processing analysis of graph perception. J. Am. Stat. Assoc., 82:454--465, Jun 1987.Google ScholarCross Ref
- I. Spence and S. Lewandowsky. Displaying proportions and percentages. Applied Cog. Psych., 5:61--77, 1991.Google ScholarCross Ref
- M. Stone and L. Bartram. Alpha, contrast and the perception of visual metadata. In Color Imaging Conf., 2009.Google Scholar
- L. Tremmel. The visual separability of plotting symbols in scatterplots. J. Comp. and Graphical Stat., 4:101--112, Jun 1995.Google Scholar
- Turkit. http://groups.csail.mit.edu/uid/turkit/.Google Scholar
- F.B. Viegas, M. Wattenberg, F. van Ham, J. Kriss, and M. McKeon. ManyEyes: A site for visualization at internet scale. IEEE TVCG, 13(6):1121--1128, 2007. Google ScholarDigital Library
- C. Ware. Information Visualization: Perception for Design. Morgan-Kaufmann, 2nd edition, 2004. Google ScholarDigital Library
- M. Wattenberg. Visualizing the stock market. In ACM CHI Extended Abstracts, pages 188--189, 1999. Google ScholarDigital Library
- D. Wigdor, C. Shen, C. Forlines, and R. Balakrishnan. Perception of elementary graphical elements in tabletop and multi-surface environments. In ACM CHI, pages 473--482, Apr 2007. Google ScholarDigital Library
- C. Ziemkiewicz and R. Kosara. The shaping of information by visual metaphors. IEEE TVCG, 14:1269--1276, Nov/Dec 2008. Google ScholarDigital Library
Index Terms
- Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Recommendations
Evaluating the accessibility of crowdsourcing tasks on Amazon's mechanical turk
ASSETS '14: Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibilityCrowd work web sites such as Amazon Mechanical Turk enable individuals to work from home, which may be useful for people with disabilities. However, the web sites for finding and performing crowd work tasks must be accessible if people with disabilities ...
Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative View
CSCWMicrotask crowdsourcing holds great potential as an employment opportunity with the flexibility and anonymity that individuals with disability may require. Though prior research has explored the accessibility of crowd work, the lived crowd work ...
Graphical Perception of Saliency-based Model Explanations
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsIn recent years, considerable work has been devoted to explaining predictive, deep learning-based models, and in turn how to evaluate explanations. An important class of evaluation methods are ones that are human-centered, which typically require the ...
Comments