VisLab: Crowdsourcing Visualization Experiments in the Wild

When creating a visualization to understand and communicate data, we face different design choices. Even though past empirical research provides foundational knowledge for visualization design, practitioners still rely on their hunches to deal with intricate trade-offs in the wild. On the other hand, researchers lack the time and resources to rigorously explore the growing design space through controlled experiments. In this work, we aim to address this two-fold problem by crowdsourcing visualization experiments. We developed VisLab, an online platform in which anyone can design and deploy experiments to evaluate their visualizations. To alleviate the complexity of experiment design and analysis, our platform provides scaffold templates and analytic dashboards. To motivate broad participation in the experiments, the platform enables anonymous participation and provides personalized performance feedback. We present use case scenarios that demonstrate the usability and usefulness of the platform in addressing the different needs of practitioners, researchers, and educators.