Recently, several information visualization (IV) tools have been produced and there is a growing number of commercial products. To contribute to a widespread adoption of IV tools, it is indispensable that these tools are effective, efficient and satisfying for the intended users. Various evaluation techniques can be considered and applied at the different phases of the IV software life-cycle. In this paper we propose an inspection technique based on the use of evaluation patterns, called Abstract Tasks, that take into account the specific nature of information visualization systems.
{"title":"Systematic inspection of information visualization systems","authors":"C. Ardito, P. Buono, M. Costabile, R. Lanzilotti","doi":"10.1145/1168149.1168163","DOIUrl":"https://doi.org/10.1145/1168149.1168163","url":null,"abstract":"Recently, several information visualization (IV) tools have been produced and there is a growing number of commercial products. To contribute to a widespread adoption of IV tools, it is indispensable that these tools are effective, efficient and satisfying for the intended users. Various evaluation techniques can be considered and applied at the different phases of the IV software life-cycle. In this paper we propose an inspection technique based on the use of evaluation patterns, called Abstract Tasks, that take into account the specific nature of information visualization systems.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"72 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125615240","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper presents an analysis of user studies from a review of papers describing new visualisation applications and uses these to highlight various issues related to the evaluation of visualisations. We first consider some of the reasons why the process of evaluating visualisations is so difficult. We then dissect the problem by discussing the importance of recognising the nature of experimental design, datasets and participants as well as the statistical analysis of results. We propose explorative evaluation as a method of discovering new things about visualisation techniques, which may give us a better understanding of the mechanisms of visualisations. Finally we give some practical guidance on how to do evaluation correctly.
{"title":"An explorative analysis of user evaluation studies in information visualisation","authors":"Geoffrey P. Ellis, A. Dix","doi":"10.1145/1168149.1168152","DOIUrl":"https://doi.org/10.1145/1168149.1168152","url":null,"abstract":"This paper presents an analysis of user studies from a review of papers describing new visualisation applications and uses these to highlight various issues related to the evaluation of visualisations. We first consider some of the reasons why the process of evaluating visualisations is so difficult. We then dissect the problem by discussing the importance of recognising the nature of experimental design, datasets and participants as well as the statistical analysis of results. We propose explorative evaluation as a method of discovering new things about visualisation techniques, which may give us a better understanding of the mechanisms of visualisations. Finally we give some practical guidance on how to do evaluation correctly.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"539 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125987217","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
T. Zuk, L. Schlesier, Petra Neumann, Mark S. Hancock, Sheelagh Carpendale
Heuristic evaluation is a well known discount evaluation technique in human-computer interaction (HCI) but has not been utilized in information visualization (InfoVis) to the same extent. While several sets of heuristics have been used or proposed for InfoVis, it is not yet known what kind of heuristics are useful for finding general InfoVis problems. We performed a meta-analysis with the goal of exploring the issues of heuristic evaluation for InfoVis. This meta-analysis concentrates on issues pertaining to the selection and organization of heuristics, and the process itself. For this purpose, we used three sets of previously published heuristics to assess a visual decision support system that is used to examine simulation data. The meta-analysis shows that the evaluation process and results have a high dependency on the heuristics and the types of evaluators chosen. We describe issues related to interpretation, redundancy, and conflict in heuristics. We also provide a discussion of generalizability and categorization of these heuristics.
{"title":"Heuristics for information visualization evaluation","authors":"T. Zuk, L. Schlesier, Petra Neumann, Mark S. Hancock, Sheelagh Carpendale","doi":"10.1145/1168149.1168162","DOIUrl":"https://doi.org/10.1145/1168149.1168162","url":null,"abstract":"Heuristic evaluation is a well known discount evaluation technique in human-computer interaction (HCI) but has not been utilized in information visualization (InfoVis) to the same extent. While several sets of heuristics have been used or proposed for InfoVis, it is not yet known what kind of heuristics are useful for finding general InfoVis problems. We performed a meta-analysis with the goal of exploring the issues of heuristic evaluation for InfoVis. This meta-analysis concentrates on issues pertaining to the selection and organization of heuristics, and the process itself. For this purpose, we used three sets of previously published heuristics to assess a visual decision support system that is used to examine simulation data. The meta-analysis shows that the evaluation process and results have a high dependency on the heuristics and the types of evaluators chosen. We describe issues related to interpretation, redundancy, and conflict in heuristics. We also provide a discussion of generalizability and categorization of these heuristics.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"79 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134072053","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Bongshin Lee, C. Plaisant, C. Parr, Jean-Daniel Fekete, N. Riche
Our goal is to define a list of tasks for graph visualization that has enough detail and specificity to be useful to: 1) designers who want to improve their system and 2) to evaluators who want to compare graph visualization systems. In this paper, we suggest a list of tasks we believe are commonly encountered while analyzing graph data. We define graph specific objects and demonstrate how all complex tasks could be seen as a series of low-level tasks performed on those objects. We believe that our taxonomy, associated with benchmark datasets and specific tasks, would help evaluators generalize results collected through a series of controlled experiments.
{"title":"Task taxonomy for graph visualization","authors":"Bongshin Lee, C. Plaisant, C. Parr, Jean-Daniel Fekete, N. Riche","doi":"10.1145/1168149.1168168","DOIUrl":"https://doi.org/10.1145/1168149.1168168","url":null,"abstract":"Our goal is to define a list of tasks for graph visualization that has enough detail and specificity to be useful to: 1) designers who want to improve their system and 2) to evaluators who want to compare graph visualization systems. In this paper, we suggest a list of tasks we believe are commonly encountered while analyzing graph data. We define graph specific objects and demonstrate how all complex tasks could be seen as a series of low-level tasks performed on those objects. We believe that our taxonomy, associated with benchmark datasets and specific tasks, would help evaluators generalize results collected through a series of controlled experiments.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123058986","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
H. Goodell, Chih-Hung Chiang, C. Kelleher, A. Baumann, G. Grinstein
To be most useful, evaluation metrics should be based on detailed observation and effective analysis of a full spectrum of system use. Because observation is costly, ideally we want a system to provide in-depth data collection with allied analyses of the key user interface elements. We have developed a visualization and analysis platform [1] that automatically records user actions and states at a high semantic level [2 and 3], and can be directly restored to any state. Audio and text annotations are collected and indexed to states, allowing users to comment on their current situation as they work, and/or as they review the session. These capabilities can be applied to usability evaluation of the system, describing problems they encountered, or to suggest improvements to the environment. Additionally, computed metrics are provided at each state [3, 4, and 5]. We believe that the metrics and the associated history data will allow us to deduce patterns of data exploration, to compare users, to evaluate tools, and to understand in a more automated approach the usability of the visualization system as a whole.
{"title":"Metrics for analyzing rich session histories","authors":"H. Goodell, Chih-Hung Chiang, C. Kelleher, A. Baumann, G. Grinstein","doi":"10.1145/1168149.1168160","DOIUrl":"https://doi.org/10.1145/1168149.1168160","url":null,"abstract":"To be most useful, evaluation metrics should be based on detailed observation and effective analysis of a full spectrum of system use. Because observation is costly, ideally we want a system to provide in-depth data collection with allied analyses of the key user interface elements. We have developed a visualization and analysis platform [1] that automatically records user actions and states at a high semantic level [2 and 3], and can be directly restored to any state. Audio and text annotations are collected and indexed to states, allowing users to comment on their current situation as they work, and/or as they review the session. These capabilities can be applied to usability evaluation of the system, describing problems they encountered, or to suggest improvements to the environment. Additionally, computed metrics are provided at each state [3, 4, and 5]. We believe that the metrics and the associated history data will allow us to deduce patterns of data exploration, to compare users, to evaluate tools, and to understand in a more automated approach the usability of the visualization system as a whole.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123098215","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The definition and usage of quality metrics for Information Visualization techniques is still an immature field. Several proposals are available but a common view and understanding of this issue is still missing. This paper attempts a first step toward a visual quality metrics systematization, providing a general classification of both metrics and usage purposes. Moreover, the paper explores a quite neglected class of visual quality metrics, namely Feature Preservation Metrics, that allow for evaluating and improving in a novel way the effectiveness of basic Infovis techniques.
{"title":"Visual quality metrics","authors":"E. Bertini, G. Santucci","doi":"10.1145/1168149.1168159","DOIUrl":"https://doi.org/10.1145/1168149.1168159","url":null,"abstract":"The definition and usage of quality metrics for Information Visualization techniques is still an immature field. Several proposals are available but a common view and understanding of this issue is still missing. This paper attempts a first step toward a visual quality metrics systematization, providing a general classification of both metrics and usage purposes. Moreover, the paper explores a quite neglected class of visual quality metrics, namely Feature Preservation Metrics, that allow for evaluating and improving in a novel way the effectiveness of basic Infovis techniques.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122092408","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This methodological note focuses on the edge density of real world examples of networks. The edge density is a parameter of interest typically when putting up user studies in an effort to prove the robustness or superiority of a novel graph visualization technique. We survey many real world examples all being of equal interest in Information Visualization, and draw a list of conclusions on how to tune edge density when randomly generating graphs in order to build artificial though realistic examples.
{"title":"Just how dense are dense graphs in the real world?: a methodological note","authors":"G. Melançon","doi":"10.1145/1168149.1168167","DOIUrl":"https://doi.org/10.1145/1168149.1168167","url":null,"abstract":"This methodological note focuses on the edge density of real world examples of networks. The edge density is a parameter of interest typically when putting up user studies in an effort to prove the robustness or superiority of a novel graph visualization technique. We survey many real world examples all being of equal interest in Information Visualization, and draw a list of conclusions on how to tune edge density when randomly generating graphs in order to build artificial though realistic examples.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129555650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Whiting, W. Cowley, J. Haack, Douglas Love, S. Tratz, Carrie Varley, Kim Wiessner
We present the Threat Stream Data Generator, an approach and tool for creating synthetic data sets for the test and evaluation of visual analytics tools and environments. We have focused on working with information analysts to understand the characteristics of threat data, to develop scenarios that will allow us to define data sets with known ground truth, to define a process of mapping threat elements in a scenario to expressions in data, and creating a software system to generate the data. We are also developing approaches to evaluating our data sets considering characteristics such as threat subtlety and appropriateness of data for the software to be examined.
{"title":"Threat stream data generator: creating the known unknowns for test and evaluation of visual analytics tools","authors":"M. Whiting, W. Cowley, J. Haack, Douglas Love, S. Tratz, Carrie Varley, Kim Wiessner","doi":"10.1145/1168149.1168166","DOIUrl":"https://doi.org/10.1145/1168149.1168166","url":null,"abstract":"We present the Threat Stream Data Generator, an approach and tool for creating synthetic data sets for the test and evaluation of visual analytics tools and environments. We have focused on working with information analysts to understand the characteristics of threat data, to develop scenarios that will allow us to define data sets with known ground truth, to define a process of mapping threat elements in a scenario to expressions in data, and creating a software system to generate the data. We are also developing approaches to evaluating our data sets considering characteristics such as threat subtlety and appropriateness of data for the software to be examined.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123061228","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
After an historical review of evaluation methods, we describe an emerging research method called Multi-dimensional In-depth Long-term Case studies (MILCs) which seems well adapted to study the creative activities that users of information visualization systems engage in. We propose that the efficacy of tools can be assessed by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users' success in achieving their professional goals. We summarize lessons from related ethnography methods used in HCI and provide guidelines for conducting MILCs for information visualization. We suggest ways to refine the methods for MILCs in modest sized projects and then envision ambitious projects with 3-10 researchers working over 1-3 years to understand individual and organizational use of information visualization by domain experts working at the frontiers of knowledge in their fields.
{"title":"Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies","authors":"B. Shneiderman, C. Plaisant","doi":"10.1145/1168149.1168158","DOIUrl":"https://doi.org/10.1145/1168149.1168158","url":null,"abstract":"After an historical review of evaluation methods, we describe an emerging research method called Multi-dimensional In-depth Long-term Case studies (MILCs) which seems well adapted to study the creative activities that users of information visualization systems engage in. We propose that the efficacy of tools can be assessed by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users' success in achieving their professional goals. We summarize lessons from related ethnography methods used in HCI and provide guidelines for conducting MILCs for information visualization. We suggest ways to refine the methods for MILCs in modest sized projects and then envision ambitious projects with 3-10 researchers working over 1-3 years to understand individual and organizational use of information visualization by domain experts working at the frontiers of knowledge in their fields.","PeriodicalId":235801,"journal":{"name":"Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123088131","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：