O. Khan, Björn þór Jónsson, Jan Zahálka, S. Rudinac, M. Worring
Interactive learning is an umbrella term for methods that attempt to understand the information need of the user and formulate queries that satisfy that information need. We propose to apply the state of the art in interactive multimodal learning to visual lifelog exploration and search, using the Exquisitor system. Exquisitor is a highly scalable interactive learning system, which uses semantic features extracted from visual content and text to suggest relevant media items to the user, based on user relevance feedback on previously suggested items. Findings from our initial experiments indicate that interactive multimodal learning will likely work well for some LSC tasks, but also suggest some potential enhancements.
{"title":"Exquisitor at the Lifelog Search Challenge 2019","authors":"O. Khan, Björn þór Jónsson, Jan Zahálka, S. Rudinac, M. Worring","doi":"10.1145/3326460.3329156","DOIUrl":"https://doi.org/10.1145/3326460.3329156","url":null,"abstract":"Interactive learning is an umbrella term for methods that attempt to understand the information need of the user and formulate queries that satisfy that information need. We propose to apply the state of the art in interactive multimodal learning to visual lifelog exploration and search, using the Exquisitor system. Exquisitor is a highly scalable interactive learning system, which uses semantic features extracted from visual content and text to suggest relevant media items to the user, based on user relevance feedback on previously suggested items. Findings from our initial experiments indicate that interactive multimodal learning will likely work well for some LSC tasks, but also suggest some potential enhancements.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127455059","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Nguyen-Khang Le, Dieu-Hien Nguyen, T. Hoang, Thanh-An Nguyen, Thanh-Dat Truong, Tung Dinh Duy, Quoc-An Luong, Viet-Khoa Vo-Ho, Vinh-Tiep Nguyen, M. Tran
Lifelogging is a new trend of research in which we collect and analyze people's daily activities to gain insight into people daily basis and enhance people's life quality. This leads to many research in retrieval systems to be developed for lifelogging analysis. One of the most important factors of a successful lifelogging retrieval system is to enhance the user's ability to perform any queries, even arbitrary ones. This motivates our proposal of a retrieval system in which we focus on the two main properties. The first property is the ability to accurately annotate the lifelog dataset with metadata that is meaningful for the retrieving process. The second property focuses on a powerful user interface that supports a novice user to perform the retrieval efficiently.
{"title":"Smart Lifelog Retrieval System with Habit-based Concepts and Moment Visualization","authors":"Nguyen-Khang Le, Dieu-Hien Nguyen, T. Hoang, Thanh-An Nguyen, Thanh-Dat Truong, Tung Dinh Duy, Quoc-An Luong, Viet-Khoa Vo-Ho, Vinh-Tiep Nguyen, M. Tran","doi":"10.1145/3326460.3329155","DOIUrl":"https://doi.org/10.1145/3326460.3329155","url":null,"abstract":"Lifelogging is a new trend of research in which we collect and analyze people's daily activities to gain insight into people daily basis and enhance people's life quality. This leads to many research in retrieval systems to be developed for lifelogging analysis. One of the most important factors of a successful lifelogging retrieval system is to enhance the user's ability to perform any queries, even arbitrary ones. This motivates our proposal of a retrieval system in which we focus on the two main properties. The first property is the ability to accurately annotate the lifelog dataset with metadata that is meaningful for the retrieving process. The second property focuses on a powerful user interface that supports a novice user to perform the retrieval efficiently.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"88 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125072074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jakub Lokoč, Tomás Soucek, Premysl Cech, Gregor Kovalcík
We present an enhanced version of the VIRET tool, an interactive video retrieval system for known-item and ad-hoc search tasks. The new features for the Lifelog Search Challenge comprise attribute-based filtering using lifelog specific modalities like time, biometric data, and GPS locations. The interface and models for content-based retrieval in image sequences are the same as for the Video Browser Showdown 2019. Hence, the performance of temporal multi-modal queries implemented for video data can be compared with other lifelog systems.
{"title":"Enhanced VIRET Tool for Lifelog Data","authors":"Jakub Lokoč, Tomás Soucek, Premysl Cech, Gregor Kovalcík","doi":"10.1145/3326460.3329159","DOIUrl":"https://doi.org/10.1145/3326460.3329159","url":null,"abstract":"We present an enhanced version of the VIRET tool, an interactive video retrieval system for known-item and ad-hoc search tasks. The new features for the Lifelog Search Challenge comprise attribute-based filtering using lifelog specific modalities like time, biometric data, and GPS locations. The interface and models for content-based retrieval in image sequences are the same as for the Video Browser Showdown 2019. Hence, the performance of temporal multi-modal queries implemented for video data can be compared with other lifelog systems.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129696493","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The semantic gap between textual queries and visual concepts is one of the key challenges in lifelog retrieval. This work presents an interactive system aimed at improving the retrieval accuracy by query term suggestion. Besides, this system also assists users to refine the retrieval results by image similarity clustering. For recommending a list of candidate words, we extract visual concepts from images by using computer vision models, and then incorporate both official and additional concepts into our system using pre-trained word embedding, in which textual knowledge is inherent. We also purpose an intelligent mechanism for rapidly removing multiple irrelevant search results. For reaching out this purpose, we build kd-trees [1] offline for reducing the computational overhead and cluster similar images by nearest neighbor search in the embedding space. Whenever users exclude some irrelevant images, their nearest neighbors in the image embedding space are also removed. In this way, users can efficiently screen out the relevant results and purge the irrelevant ones, scanning over more retrieval results in a shorter period of time.
{"title":"An Interactive Approach to Integrating External Textual Knowledge for Multimodal Lifelog Retrieval","authors":"Chia-Chun Chang, Min-Huan Fu, Hen-Hsen Huang, Hsin-Hsi Chen","doi":"10.1145/3326460.3329163","DOIUrl":"https://doi.org/10.1145/3326460.3329163","url":null,"abstract":"The semantic gap between textual queries and visual concepts is one of the key challenges in lifelog retrieval. This work presents an interactive system aimed at improving the retrieval accuracy by query term suggestion. Besides, this system also assists users to refine the retrieval results by image similarity clustering. For recommending a list of candidate words, we extract visual concepts from images by using computer vision models, and then incorporate both official and additional concepts into our system using pre-trained word embedding, in which textual knowledge is inherent. We also purpose an intelligent mechanism for rapidly removing multiple irrelevant search results. For reaching out this purpose, we build kd-trees [1] offline for reducing the computational overhead and cluster similar images by nearest neighbor search in the embedding space. Whenever users exclude some irrelevant images, their nearest neighbors in the image embedding space are also removed. In this way, users can efficiently screen out the relevant results and purge the irrelevant ones, scanning over more retrieval results in a shorter period of time.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114474206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. P. Nguyen, Dien H. Le, U. Pham, M. Crane, G. Healy, C. Gurrin
With the appearance of many wearable devices like smartwatches, recording glasses (such as Google glass), smart phones, digital personal profiles have become more readily available nowadays. However, searching and navigating these multi-source, multi-modal, and often unstructured data to extract useful information is still a relatively challenging task. Therefore, the LSC2019 competition has been organized so that researchers can demonstrate novel search engines, as well as exchange ideas and collaborate on these types of problems. We present in this paper our approach for supporting interactive searches of lifelog data by employing a new retrieval system calledVieLens, which is an interactive retrieval system enhanced by natural language processing techniques to extend and improve search results mainly in the context of a user's activities in their daily life.
{"title":"VieLens,: An Interactive Search engine for LSC2019","authors":"S. P. Nguyen, Dien H. Le, U. Pham, M. Crane, G. Healy, C. Gurrin","doi":"10.1145/3326460.3329161","DOIUrl":"https://doi.org/10.1145/3326460.3329161","url":null,"abstract":"With the appearance of many wearable devices like smartwatches, recording glasses (such as Google glass), smart phones, digital personal profiles have become more readily available nowadays. However, searching and navigating these multi-source, multi-modal, and often unstructured data to extract useful information is still a relatively challenging task. Therefore, the LSC2019 competition has been organized so that researchers can demonstrate novel search engines, as well as exchange ideas and collaborate on these types of problems. We present in this paper our approach for supporting interactive searches of lifelog data by employing a new retrieval system calledVieLens, which is an interactive retrieval system enhanced by natural language processing techniques to extend and improve search results mainly in the context of a user's activities in their daily life.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117188526","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Isadora Nguyen Van Khan, Pranita Shrestha, Min Zhang, Yiqun Liu, Shaoping Ma
The rise of affordable wearable devices makes visual lifelog data easier to collect. This data can then be retrieved as an aside memory usable as well for people with memory issues or in daily life to be more efficient. This paper presents the retrieval system designed for the Lifelog Search Challenge 2019 competition. This retrieval system offers two search levels: a semantic one and an image-based one. The data used for the indexing for both levels (semantic data for the text-based method or combinations of HSV Histogram and BRIEF features for the example-based one) as well as the architecture of the overall system is described. Results from our experiments are promising. The text-based search engine has disparate results that strongly depend on the keywords used for the query. For the example search engine, the best features are composed of both HSV histogram and BRIEF features. Despite encouraging results, the processing time appears to be a limitation.
{"title":"A Two-Level Lifelog Search Engine at the LSC 2019","authors":"Isadora Nguyen Van Khan, Pranita Shrestha, Min Zhang, Yiqun Liu, Shaoping Ma","doi":"10.1145/3326460.3329158","DOIUrl":"https://doi.org/10.1145/3326460.3329158","url":null,"abstract":"The rise of affordable wearable devices makes visual lifelog data easier to collect. This data can then be retrieved as an aside memory usable as well for people with memory issues or in daily life to be more efficient. This paper presents the retrieval system designed for the Lifelog Search Challenge 2019 competition. This retrieval system offers two search levels: a semantic one and an image-based one. The data used for the indexing for both levels (semantic data for the text-based method or combinations of HSV Histogram and BRIEF features for the example-based one) as well as the architecture of the overall system is described. Results from our experiments are promising. The text-based search engine has disparate results that strongly depend on the keywords used for the query. For the example search engine, the best features are composed of both HSV histogram and BRIEF features. Despite encouraging results, the processing time appears to be a limitation.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126705327","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Andreas Leibetseder, Bernd Münzer, Manfred Jürgen Primus, Sabrina Kletz, Klaus Schöffmann, Fabian Berns, C. Beecks
First held in Yokohama, Japan in 2018, the Lifelogging Search Challenge (LSC) is an interactive lifelogging content retrieval competition, where international teams are faced with the task of finding specific moments within a large collection of multimodal lifelogging data. As one of the successfully participating LSC2018 teams (2nd place), we present our improved lifeXplore system for the second iteration of this challenge. Based on diveXplore, an interactive video retrieval system that has been developed for the annual Video Browser Showdown, the lifeXplore system similarly operates on shot-segmented video data, while maintaining browseable feature maps and offering customizable similarity search, yet also introducing additional features such as metadata filtering and interactive day inspection. Alongside several smaller functionality and usability improvements, the LSC2019 version of the system in particular includes combinable filters, geolocation-based filtering and automatic feature map browsing in order to improve a user's searching and browsing experience.
{"title":"lifeXplore at the Lifelog Search Challenge 2019","authors":"Andreas Leibetseder, Bernd Münzer, Manfred Jürgen Primus, Sabrina Kletz, Klaus Schöffmann, Fabian Berns, C. Beecks","doi":"10.1145/3326460.3329157","DOIUrl":"https://doi.org/10.1145/3326460.3329157","url":null,"abstract":"First held in Yokohama, Japan in 2018, the Lifelogging Search Challenge (LSC) is an interactive lifelogging content retrieval competition, where international teams are faced with the task of finding specific moments within a large collection of multimodal lifelogging data. As one of the successfully participating LSC2018 teams (2nd place), we present our improved lifeXplore system for the second iteration of this challenge. Based on diveXplore, an interactive video retrieval system that has been developed for the annual Video Browser Showdown, the lifeXplore system similarly operates on shot-segmented video data, while maintaining browseable feature maps and offering customizable similarity search, yet also introducing additional features such as metadata filtering and interactive day inspection. Alongside several smaller functionality and usability improvements, the LSC2019 version of the system in particular includes combinable filters, geolocation-based filtering and automatic feature map browsing in order to improve a user's searching and browsing experience.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133479175","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, M. Tran, Liting Zhou, Pablo Redondo, Sinéad Smyth, C. Gurrin
In this paper we present a first version of the LifeSeeker interactive lifelog retrieval engine that is under development at Dublin City University. This retrieval engine has been designed as a platform onto which future lifelog annotation and retrieval engines will be built. The first implementation of LifeSeeker has been designed for the LSC'19 comparative benchmarking challenge and it takes the form of a faceted search and browsing interface with the addition of query expansion to help solve the lexical-gap between novice users and the concept annotation tools employed for annotating the collection.
{"title":"LifeSeeker: Interactive Lifelog Search Engine at LSC 2019","authors":"Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, M. Tran, Liting Zhou, Pablo Redondo, Sinéad Smyth, C. Gurrin","doi":"10.1145/3326460.3329162","DOIUrl":"https://doi.org/10.1145/3326460.3329162","url":null,"abstract":"In this paper we present a first version of the LifeSeeker interactive lifelog retrieval engine that is under development at Dublin City University. This retrieval engine has been designed as a platform onto which future lifelog annotation and retrieval engines will be built. The first implementation of LifeSeeker has been designed for the LSC'19 comparative benchmarking challenge and it takes the form of a faceted search and browsing interface with the addition of query expansion to help solve the lexical-gap between novice users and the concept annotation tools employed for annotating the collection.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125420807","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Parian, H. Schuldt
With the increase in sensory capability of mobile devices, the data that can be generated and used in a lifelogging context gets increasingly diverse. Such data is special in the context of multimedia, not only because of its close personal relationship with its originator, but also because of its diverse multimodality and its composition from structured, semi-structured, and unstructured data. This diversity poses retrieval challenges that are unique to lifelog data but which also have implications for retrieval activity in other multimedia domains. In this paper, we present the extensions made to the vitrivr open-source multimedia retrieval stack, in order to address some of these unique lifelogging challenges. For the participation to the 2019 Lifelog Search Challenge (LSC), we have extended vitrivr with the capability to process Boolean query expressions alongside content-based query descriptions in order to leverage the structural diversity inherent to lifelog data.
{"title":"Retrieval of Structured and Unstructured Data with vitrivr","authors":"Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Parian, H. Schuldt","doi":"10.1145/3326460.3329160","DOIUrl":"https://doi.org/10.1145/3326460.3329160","url":null,"abstract":"With the increase in sensory capability of mobile devices, the data that can be generated and used in a lifelogging context gets increasingly diverse. Such data is special in the context of multimedia, not only because of its close personal relationship with its originator, but also because of its diverse multimodality and its composition from structured, semi-structured, and unstructured data. This diversity poses retrieval challenges that are unique to lifelog data but which also have implications for retrieval activity in other multimedia domains. In this paper, we present the extensions made to the vitrivr open-source multimedia retrieval stack, in order to address some of these unique lifelogging challenges. For the participation to the 2019 Lifelog Search Challenge (LSC), we have extended vitrivr with the capability to process Boolean query expressions alongside content-based query descriptions in order to leverage the structural diversity inherent to lifelog data.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"23 9","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132969178","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the ACM Workshop on Lifelog Search Challenge","authors":"","doi":"10.1145/3326460","DOIUrl":"https://doi.org/10.1145/3326460","url":null,"abstract":"","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133994085","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：