Games with a Purpose: Harnessing Human Computation for Robust Evaluation of Visualization

Abstract: Due to the inherent characteristics of the visualization process, most of the problems in this field have strong ties with human cognition and perception. This makes the human brain and sensory system the only appropriate platform for evaluating and fine-tuning a new visualization method or paradigm. However, getting humans to volunteer for these purposes has always been a significant obstacle, and thus this phase of the development process has traditionally formed a bottleneck, slowing down progress in visualization research. We propose to take advantage of the newly emerging field of Human Computation (HC) to overcome these challenges. HC promotes the idea that rather than considering humans as users of the computational system, they can be made part of a hybrid computational loop consisting of traditional computation resources and the human brain and sensory system. This approach is particularly successful in cases where part of the computational problem is considered intractable using known computer algorithms but is trivial to common sense human knowledge. In this paper, we focus on HC from the perspective of solving visualization problems and also outline a framework by which humans can be easily seduced to volunteer their HC resources. We introduce a purpose-driven game entitled “Disguise” which serves as a prototypical example for how the evaluation of visualization algorithms can be mapped into a fun and addictive activity, allowing this task to be accomplished in an extensive yet cost effective way. Finally, we sketch out a framework that transcends from the pure evaluation of existing visualization methods to the design of new ones.

Teaser: Concept of our purpose-driven game designed to evaluate three published and one new color blending algorithm.

(Left) Illustration of how an intruder moves randomly over a collector before self-explosion. (Right) A side-view of the game environment showing the three layers. The collectors are always in the middle layer while intruders can float around either on the top or the bottom layer. The markers A, B, C show three possible action scenarios from the player. Case A: The intruder is a layer below the collector. Striking here hits the collector and damages it. Case B: The intruder is a layer above the collector. Striking here disables the intruder keeping the collector intact. Also, since it is on top of the collector, the player scores a point. Case C: The intruder is in either layer but outside the collector radius. Striking here disables the intruder but no points are scored or damage is done to the collector.

Video: Watch to see some scenes of the game.

Extended project page: Visit this page where you can also find the link to the game itself.

Paper: N. Ahmed, Z. Zheng, K. Mueller, "Human Computation in Visualization: Using Purpose Driven Games for Robust Evaluation of Visualization Algorithms," IEEE Transactions on Visualization and Computer Graphics, 18(12): 2104-2113, 2012. pdf ppt

Funding: NSF grant IIS 1117132 and Department of Energy Award DEOE0000220.