We propose a set of common sense steps required to develop a recommender system for visual analytics. Such a system is an essential way to get additional mileage out of costly user studies, which are typically archived post publication. Crucially, we propose conducting user studies in a manner that allows machine learning techniques to elucidate relationships between experimental data (i.e., user performance) and metrics about the data being visualized and candidate visual representations. We execute a case study within our framework to extract simple rules of thumb that relate different data metrics and visualization characteristics to patterns of user errors on several network analysis tasks. Our case study suggests a research agenda supporting the development of general, robust visualization recommender systems.
{"title":"More bang for your research buck: toward recommender systems for visual analytics","authors":"L. Blaha, Dustin L. Arendt, Fairul Mohd-Zaid","doi":"10.1145/2669557.2669566","DOIUrl":"https://doi.org/10.1145/2669557.2669566","url":null,"abstract":"We propose a set of common sense steps required to develop a recommender system for visual analytics. Such a system is an essential way to get additional mileage out of costly user studies, which are typically archived post publication. Crucially, we propose conducting user studies in a manner that allows machine learning techniques to elucidate relationships between experimental data (i.e., user performance) and metrics about the data being visualized and candidate visual representations. We execute a case study within our framework to extract simple rules of thumb that relate different data metrics and visualization characteristics to patterns of user errors on several network analysis tasks. Our case study suggests a research agenda supporting the development of general, robust visualization recommender systems.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131552571","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
To shed more light on data explorers dealing with complex information visualizations in real world scenarios, new methodologies and models are needed which overcome existing explanatory gaps. Therefore, a novel model to analyze users' errors and insights is outlined that is derived from Rasmussen's model on different levels of cognitive processing, and integrates explorers' skills, schemes, and knowledge. After locating this model in the landscape of theories for visual analytics, the main building blocks of the model, where three cognitive processing levels are interlinked, are described in detail. Finally, its applicability, challenges in measurement and future research options are discussed.
{"title":"Just the other side of the coin?: from error- to insight-analysis","authors":"M. Smuc","doi":"10.1145/2669557.2669570","DOIUrl":"https://doi.org/10.1145/2669557.2669570","url":null,"abstract":"To shed more light on data explorers dealing with complex information visualizations in real world scenarios, new methodologies and models are needed which overcome existing explanatory gaps. Therefore, a novel model to analyze users' errors and insights is outlined that is derived from Rasmussen's model on different levels of cognitive processing, and integrates explorers' skills, schemes, and knowledge. After locating this model in the landscape of theories for visual analytics, the main building blocks of the model, where three cognitive processing levels are interlinked, are described in detail. Finally, its applicability, challenges in measurement and future research options are discussed.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132172862","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dimension Reduction techniques used to visualize multidimensional data provide a scatterplot spatialization of data similarities. A widespread way to evaluate the quality of such DR techniques is to use labeled data as a ground truth and to call the reader as a witness to qualify the visualization by looking at class-cluster correlations within the scatterplot. We expose the pitfalls of this evaluation process and we propose a principled solution to guide researchers to improve the way they use this visual evaluation of DR techniques.
{"title":"Sanity check for class-coloring-based evaluation of dimension reduction techniques","authors":"Michaël Aupetit","doi":"10.1145/2669557.2669578","DOIUrl":"https://doi.org/10.1145/2669557.2669578","url":null,"abstract":"Dimension Reduction techniques used to visualize multidimensional data provide a scatterplot spatialization of data similarities. A widespread way to evaluate the quality of such DR techniques is to use labeled data as a ground truth and to call the reader as a witness to qualify the visualization by looking at class-cluster correlations within the scatterplot. We expose the pitfalls of this evaluation process and we propose a principled solution to guide researchers to improve the way they use this visual evaluation of DR techniques.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116894669","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Eye tracking can be a suitable evaluation method for determining which regions and objects of a stimulus a human viewer perceived. Analysts can use eye tracking as a complement to other evaluation methods for a more holistic assessment of novel visualization techniques beyond time and error measures. Up to now, most stimuli in eye tracking are either static stimuli or videos. Since interaction is an integral part of visualization, an evaluation should include interaction. In this paper, we present an extensive literature review on evaluation methods for interactive visualizations. Based on the literature review we propose ideas for analyzing eye movement data from interactive stimuli. This requires looking critically at challenges induced by interactive stimuli. The first step is to collect data using different study methods. In our case, we look at using eye tracking, interaction logs, and thinking-aloud protocols. In addition, this requires a thorough synchronization of the mentioned study methods. To analyze the collected data new analysis techniques have to be developed. We investigate existing approaches and how we can adapt them to new data types as well as sketch ideas how new approaches can look like.
{"title":"Towards analyzing eye tracking data for evaluating interactive visualization systems","authors":"Tanja Blascheck, T. Ertl","doi":"10.1145/2669557.2669569","DOIUrl":"https://doi.org/10.1145/2669557.2669569","url":null,"abstract":"Eye tracking can be a suitable evaluation method for determining which regions and objects of a stimulus a human viewer perceived. Analysts can use eye tracking as a complement to other evaluation methods for a more holistic assessment of novel visualization techniques beyond time and error measures. Up to now, most stimuli in eye tracking are either static stimuli or videos. Since interaction is an integral part of visualization, an evaluation should include interaction. In this paper, we present an extensive literature review on evaluation methods for interactive visualizations. Based on the literature review we propose ideas for analyzing eye movement data from interactive stimuli. This requires looking critically at challenges induced by interactive stimuli. The first step is to collect data using different study methods. In our case, we look at using eye tracking, interaction logs, and thinking-aloud protocols. In addition, this requires a thorough synchronization of the mentioned study methods. To analyze the collected data new analysis techniques have to be developed. We investigate existing approaches and how we can adapt them to new data types as well as sketch ideas how new approaches can look like.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120940303","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Abdul-Rahman, Karl J. Proctor, Brian Duffy, Min Chen
Crowdsourcing platforms, such as Amazon's Mechanical Turk (MTurk), are providing visualization researchers with a new avenue for conducting empirical studies. While such platforms offer several advantages over lab-based studies, they also feature some "unknown" or "uncontrolled" variables, which could potentially introduce serious confounding effects in the resultant measurement data. In this paper, we present our experience of using repeated measures in three empirical studies using MTurk. Each study presented participants with a set of stimuli, each featuring a condition of an independent variable. Participants were exposed to stimuli repeatedly in a pseudo-random order through four trials and their responses were measured digitally. Only a small portion of the participants were able to perform with absolute consistency for all stimuli throughout each experiment. This suggests that a repeated measures design is highly desirable (if not essential) when designing empirical studies for crowdsourcing platforms. Additionally, the majority of participants performed their tasks with reasonable consistency when all stimuli in an experiment are considered collectively. In other words, to most participants, inconsistency occurred occasionally. This suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of "unknown" or "uncontrolled" variables in the design of the experiment.
众包平台，如亚马逊的Mechanical Turk (MTurk)，为可视化研究人员提供了进行实证研究的新途径。虽然这些平台比实验室研究有很多优势，但它们也有一些“未知”或“不受控制”的变量，这可能会在最终的测量数据中引入严重的混淆效应。在本文中，我们介绍了我们使用MTurk在三个实证研究中使用重复测量的经验。每项研究都向参与者提供一组刺激，每个刺激都有一个自变量的条件。通过四次试验，参与者以伪随机顺序反复暴露于刺激下，他们的反应被数字化测量。只有一小部分参与者能够在每次实验中对所有刺激都保持绝对一致的表现。这表明，在为众包平台设计实证研究时，重复测量设计是非常可取的(如果不是必要的)。此外，当实验中的所有刺激被集体考虑时，大多数参与者以合理的一致性执行任务。换句话说，对大多数参与者来说，不一致偶尔会发生。这表明，众包仍然是一个有效的实验环境，只要人们可以在实验设计中整合观察和减轻“未知”或“不受控制”变量的潜在混淆效应的手段。
{"title":"Repeated measures design in crowdsourcing-based experiments for visualization","authors":"A. Abdul-Rahman, Karl J. Proctor, Brian Duffy, Min Chen","doi":"10.1145/2669557.2669561","DOIUrl":"https://doi.org/10.1145/2669557.2669561","url":null,"abstract":"Crowdsourcing platforms, such as Amazon's Mechanical Turk (MTurk), are providing visualization researchers with a new avenue for conducting empirical studies. While such platforms offer several advantages over lab-based studies, they also feature some \"unknown\" or \"uncontrolled\" variables, which could potentially introduce serious confounding effects in the resultant measurement data. In this paper, we present our experience of using repeated measures in three empirical studies using MTurk. Each study presented participants with a set of stimuli, each featuring a condition of an independent variable. Participants were exposed to stimuli repeatedly in a pseudo-random order through four trials and their responses were measured digitally. Only a small portion of the participants were able to perform with absolute consistency for all stimuli throughout each experiment. This suggests that a repeated measures design is highly desirable (if not essential) when designing empirical studies for crowdsourcing platforms. Additionally, the majority of participants performed their tasks with reasonable consistency when all stimuli in an experiment are considered collectively. In other words, to most participants, inconsistency occurred occasionally. This suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of \"unknown\" or \"uncontrolled\" variables in the design of the experiment.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127934238","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alvin E Tarrell, Ann L. Fruhling, R. Borgo, C. Forsell, G. Grinstein, J. Scholtz
This position paper describes heuristic evaluation as it relates to visualization and visual analytics. We review heuristic evaluation in general, then comment on previous process-based, performance-based, and framework-based efforts to adapt the method to visualization-specific needs. We postulate that the framework-based approach holds the most promise for future progress in development of visualization-specific heuristics, and propose a specific framework as a starting point. We then recommend a method for community involvement and input into the further development of the heuristic framework and more detailed design and evaluation guidelines.
{"title":"Toward visualization-specific heuristic evaluation","authors":"Alvin E Tarrell, Ann L. Fruhling, R. Borgo, C. Forsell, G. Grinstein, J. Scholtz","doi":"10.1145/2669557.2669580","DOIUrl":"https://doi.org/10.1145/2669557.2669580","url":null,"abstract":"This position paper describes heuristic evaluation as it relates to visualization and visual analytics. We review heuristic evaluation in general, then comment on previous process-based, performance-based, and framework-based efforts to adapt the method to visualization-specific needs. We postulate that the framework-based approach holds the most promise for future progress in development of visualization-specific heuristics, and propose a specific framework as a starting point. We then recommend a method for community involvement and input into the further development of the heuristic framework and more detailed design and evaluation guidelines.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114277073","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. Scholtz, Oriana Love, M. Whiting, Duncan Hodges, Lia Emanuel, D. Fraser
In this paper, we present three case studies of utility evaluations of underlying models in software systems: a user-model, technical and social models both singly and in combination, and a research-based model for user identification. Each of the three cases used a different approach to evaluating the model and each had challenges to overcome in designing and implementing the evaluation. We describe the methods we used and challenges faced in designing the evaluation procedures, summarize the lessons learned, enumerate considerations for those undertaking such evaluations, and present directions for future work.
{"title":"Utility evaluation of models","authors":"J. Scholtz, Oriana Love, M. Whiting, Duncan Hodges, Lia Emanuel, D. Fraser","doi":"10.1145/2669557.2669562","DOIUrl":"https://doi.org/10.1145/2669557.2669562","url":null,"abstract":"In this paper, we present three case studies of utility evaluations of underlying models in software systems: a user-model, technical and social models both singly and in combination, and a research-based model for user identification. Each of the three cases used a different approach to evaluating the model and each had challenges to overcome in designing and implementing the evaluation. We describe the methods we used and challenges faced in designing the evaluation procedures, summarize the lessons learned, enumerate considerations for those undertaking such evaluations, and present directions for future work.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129461773","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Rind, W. Aigner, Markus Wagner, S. Miksch, T. Lammarsch
User tasks play a pivotal role in evaluation throughout visualization design and development. However, the term 'task' is used ambiguously within the visualization community. In this position paper, we critically analyze the relevant literature and systematically compare definitions for 'task' and the usage of related terminology. In doing so, we identify a three-dimensional conceptual space of user tasks in visualization. Using these dimensions, visualization researchers can better formulate their contributions which helps advance visualization as a whole.
{"title":"User tasks for evaluation: untangling the terminology throughout visualization design and development","authors":"A. Rind, W. Aigner, Markus Wagner, S. Miksch, T. Lammarsch","doi":"10.1145/2669557.2669568","DOIUrl":"https://doi.org/10.1145/2669557.2669568","url":null,"abstract":"User tasks play a pivotal role in evaluation throughout visualization design and development. However, the term 'task' is used ambiguously within the visualization community. In this position paper, we critically analyze the relevant literature and systematically compare definitions for 'task' and the usage of related terminology. In doing so, we identify a three-dimensional conceptual space of user tasks in visualization. Using these dimensions, visualization researchers can better formulate their contributions which helps advance visualization as a whole.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115894333","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The paper originates from the idea that in the field of information visualization, positive user experience is extremely important if we wish to see users adopt and engage with the novel information visualization tools. Suggesting the use of product reaction card method to evaluate user experience, the paper gives an example of FrbrVis prototype to demonstrate how the results of this method could be analysed and used for comparing different designs. The authors also propose five dimensions of user experience (UX) that could be gathered from reaction cards and conclude that the results from reaction cards mirror and add to other performance and preference indicators.
{"title":"Evaluation of information visualization techniques: analysing user experience with reaction cards","authors":"Tanja Mercun","doi":"10.1145/2669557.2669565","DOIUrl":"https://doi.org/10.1145/2669557.2669565","url":null,"abstract":"The paper originates from the idea that in the field of information visualization, positive user experience is extremely important if we wish to see users adopt and engage with the novel information visualization tools. Suggesting the use of product reaction card method to evaluate user experience, the paper gives an example of FrbrVis prototype to demonstrate how the results of this method could be analysed and used for comparing different designs. The authors also propose five dimensions of user experience (UX) that could be gathered from reaction cards and conclude that the results from reaction cards mirror and add to other performance and preference indicators.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122798829","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Existing evaluations of data visualizations often employ a series of low-level, detailed questions to be answered or benchmark tasks to be performed. While that methodology can be helpful to determine a visualization's usability, such evaluations overlook the key benefits that visualization uniquely provides over other data analysis methods. I propose a value-driven evaluation of visualizations in which a person illustrates a system's value through four important capabilities: minimizing the time to answer diverse questions, spurring the generation of insights and insightful questions, conveying the essence of the data, and generating confidence and knowledge about the data's domain and context. Additionally, I explain how interaction is instrumental in creating much of the value that can be found in visualizations.
{"title":"Value-driven evaluation of visualizations","authors":"J. Stasko","doi":"10.1145/2669557.2669579","DOIUrl":"https://doi.org/10.1145/2669557.2669579","url":null,"abstract":"Existing evaluations of data visualizations often employ a series of low-level, detailed questions to be answered or benchmark tasks to be performed. While that methodology can be helpful to determine a visualization's usability, such evaluations overlook the key benefits that visualization uniquely provides over other data analysis methods. I propose a value-driven evaluation of visualizations in which a person illustrates a system's value through four important capabilities: minimizing the time to answer diverse questions, spurring the generation of insights and insightful questions, conveying the essence of the data, and generating confidence and knowledge about the data's domain and context. Additionally, I explain how interaction is instrumental in creating much of the value that can be found in visualizations.","PeriodicalId":179584,"journal":{"name":"Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2014-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115080941","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：