Ly-Duyen Tran, Binh T. Nguyen, Liting Zhou, C. Gurrin
Retrieval is a fundamental challenge within the research community of lifelog and the Lifelog Search Challenge (LSC) has been an important annual benchmarking activity for interactive lifelog retrieval systems since 2018. This paper proposes MyEachtra (/mai-AK-truh/), a system designed for the upcoming LSC’23 workshop. Improved upon MyScéal, which was the top performing system from LSC’20 to LSC’22, MyEachtra includes modifications to address the challenges of non-owner user understanding of lifelog contexts and open-ended lifelog question answering. Specifically, MyEachtra shifts the focus from images to events as retrieval units. Events are segmented using location metadata as well as visual and time differences between successive images. A pilot study on different approaches to aggregate images into events was conducted to test the automatic performance of the system, which showed promising results. For known-item queries, showing only the top 3 events proved to be adequate to find relevant images. However, future evaluation of the performance for ad-hoc and question-answering queries is necessary for a complete analysis of the MyEachtra.
{"title":"MyEachtra: Event-Based Interactive Lifelog Retrieval System for LSC’23","authors":"Ly-Duyen Tran, Binh T. Nguyen, Liting Zhou, C. Gurrin","doi":"10.1145/3592573.3593100","DOIUrl":"https://doi.org/10.1145/3592573.3593100","url":null,"abstract":"Retrieval is a fundamental challenge within the research community of lifelog and the Lifelog Search Challenge (LSC) has been an important annual benchmarking activity for interactive lifelog retrieval systems since 2018. This paper proposes MyEachtra (/mai-AK-truh/), a system designed for the upcoming LSC’23 workshop. Improved upon MyScéal, which was the top performing system from LSC’20 to LSC’22, MyEachtra includes modifications to address the challenges of non-owner user understanding of lifelog contexts and open-ended lifelog question answering. Specifically, MyEachtra shifts the focus from images to events as retrieval units. Events are segmented using location metadata as well as visual and time differences between successive images. A pilot study on different approaches to aggregate images into events was conducted to test the automatic performance of the system, which showed promising results. For known-item queries, showing only the top 3 events proved to be adequate to find relevant images. However, future evaluation of the performance for ad-hoc and question-answering queries is necessary for a complete analysis of the MyEachtra.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"87 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133036739","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Quang-Linh Tran, Ly-Duyen Tran, Binh T. Nguyen, C. Gurrin
Lifelogging is an activity of recording all events that happen in the daily life of an individual. The events can contain images, audio, health index, etc which are collected through various devices such as wearable cameras, smartwatches, and other digital services. Exploiting lifelog data can bring significant benefits for lifeloggers from creating personalized healthcare plans to retrieving events in the past. In recent years, there has been a growing development of interactive lifelog retrieval systems, such as competitors at the annual Lifelog Search Challenge (LSC), to assist lifeloggers in finding events from the past. This paper introduces an interactive lifelog image retrieval called MemoriEase for the LSC’23 challenge. This system combines concept-based and embedding-based retrieval approaches to answer accurate images for LSC’23 queries. This system uses BLIP for the embedding-based retrieval approach to reduce the semantic gap between images and text queries. The concept-based retrieval approach uses full-text search in Elasticsearch to retrieve images having visual concepts similar to keywords in the query. Regarding the user interface, we make it as simple as possible to make novices users can use it with only a small effort. This is the first version of MemoriEase and we expect this can help users perform well in the LSC’23 competition.
{"title":"MemoriEase: An Interactive Lifelog Retrieval System for LSC’23","authors":"Quang-Linh Tran, Ly-Duyen Tran, Binh T. Nguyen, C. Gurrin","doi":"10.1145/3592573.3593101","DOIUrl":"https://doi.org/10.1145/3592573.3593101","url":null,"abstract":"Lifelogging is an activity of recording all events that happen in the daily life of an individual. The events can contain images, audio, health index, etc which are collected through various devices such as wearable cameras, smartwatches, and other digital services. Exploiting lifelog data can bring significant benefits for lifeloggers from creating personalized healthcare plans to retrieving events in the past. In recent years, there has been a growing development of interactive lifelog retrieval systems, such as competitors at the annual Lifelog Search Challenge (LSC), to assist lifeloggers in finding events from the past. This paper introduces an interactive lifelog image retrieval called MemoriEase for the LSC’23 challenge. This system combines concept-based and embedding-based retrieval approaches to answer accurate images for LSC’23 queries. This system uses BLIP for the embedding-based retrieval approach to reduce the semantic gap between images and text queries. The concept-based retrieval approach uses full-text search in Elasticsearch to retrieve images having visual concepts similar to keywords in the query. Regarding the user interface, we make it as simple as possible to make novices users can use it with only a small effort. This is the first version of MemoriEase and we expect this can help users perform well in the LSC’23 competition.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129519858","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
N. Hoang-Xuan, Thang-Long Nguyen-Ho, C. Gurrin, Minh-Triet Tran
AI-assisted tools have become more prevalent than ever in the last few years. However, applying them to build a lifelog retrieval system is still non-trivial due to the disparity in interfaces and interactions. The Lifelog Search Challenge (LSC) aims to provide a testing ground where systems can be benchmarked in a highly competitive setting. In this paper, we present the fourth iteration of our participating system FIRST. For this year, we adopt generative models to equip the system with predictive ability rather than entirely relying on the user to input the query. We also index a sequence of images as an event for improved search speed. Finally, we demonstrate how the additional features can assist users in searching.
{"title":"Lifelog Discovery Assistant: Suggesting Prompts and Indexing Event Sequences for FIRST at LSC 2023","authors":"N. Hoang-Xuan, Thang-Long Nguyen-Ho, C. Gurrin, Minh-Triet Tran","doi":"10.1145/3592573.3593104","DOIUrl":"https://doi.org/10.1145/3592573.3593104","url":null,"abstract":"AI-assisted tools have become more prevalent than ever in the last few years. However, applying them to build a lifelog retrieval system is still non-trivial due to the disparity in interfaces and interactions. The Lifelog Search Challenge (LSC) aims to provide a testing ground where systems can be benchmarked in a highly competitive setting. In this paper, we present the fourth iteration of our participating system FIRST. For this year, we adopt generative models to equip the system with predictive ability rather than entirely relying on the user to input the query. We also index a sequence of images as an event for improved search speed. Finally, we demonstrate how the additional features can assist users in searching.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123430160","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Maria Tysse Hordvik, Julie Sophie Teilstad Østby, M. Kesavulu, Thao-Nhu Nguyen, Tu-Khiem Le, Duc-Tien Dang-Nguyen
One of the important components of the lifelog systems is the user interface which provides the ability to quickly and easily find a specific image or set of images. Although lifelogging is a mature field in the information retrieval domain, the focus on user interfaces is not explored extensively. We start by identifying the common issues with existing lifelog systems from the user interface and user experience perspective. Following the exploration, we present a set of guidelines for designing a user interface for Lifelog systems. We introduce LifeLens- a novel minimalist user interface design specifically designed to improve the usability and ease of use of an interactive lifelog system. The initial version of the LifeLens system provides several improvements over existing lifelog systems addressing the design issues identified during the exploration. The proposed system presents several features that not only enable the users of the system to easily navigate the interface with minimal effort on the user’s part to learn and understand the features offered but also provide a minimal way to gather user feedback.
{"title":"LifeLens: Transforming Lifelog Search with Innovative UX/UI Design","authors":"Maria Tysse Hordvik, Julie Sophie Teilstad Østby, M. Kesavulu, Thao-Nhu Nguyen, Tu-Khiem Le, Duc-Tien Dang-Nguyen","doi":"10.1145/3592573.3593096","DOIUrl":"https://doi.org/10.1145/3592573.3593096","url":null,"abstract":"One of the important components of the lifelog systems is the user interface which provides the ability to quickly and easily find a specific image or set of images. Although lifelogging is a mature field in the information retrieval domain, the focus on user interfaces is not explored extensively. We start by identifying the common issues with existing lifelog systems from the user interface and user experience perspective. Following the exploration, we present a set of guidelines for designing a user interface for Lifelog systems. We introduce LifeLens- a novel minimalist user interface design specifically designed to improve the usability and ease of use of an interactive lifelog system. The initial version of the LifeLens system provides several improvements over existing lifelog systems addressing the design issues identified during the exploration. The proposed system presents several features that not only enable the users of the system to easily navigate the interface with minimal effort on the user’s part to learn and understand the features offered but also provide a minimal way to gather user feedback.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121028987","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we introduce Voxento 4.0 – an interactive voice-based retrieval system for lifelogs which has been developed to participate in the sixth Lifelog Search Challenge LSC’23, at ACM ICMR’23. Voxento has participated three times in the LSC editions and achieved the rank of 4th in LSC21 and 5th in LSC22 respectively. In this version, Voxento 4.0, we have focused on improving the previous system’s interface, voice interaction and retrieval functionality. The current version has implemented some processing and cleaning of the dataset and employs the CLIP model to extract image features. In addition, the system’s interface was redesigned for better visualisation of the elements and the images for effective interaction. This improvement in the interface will help to support voice interaction in future work. The interface developments include logging voice interaction and images displayed, submitted, selected and starred to enhance user experience with the system. The voice interaction part has also been enhanced in the workflow of the voice lifecycle interaction and with additional voice commands.
{"title":"Voxento 4.0: A More Flexible Visualisation and Control for Lifelogs","authors":"Ahmed Alateeq, M. Roantree, C. Gurrin","doi":"10.1145/3592573.3593097","DOIUrl":"https://doi.org/10.1145/3592573.3593097","url":null,"abstract":"In this paper, we introduce Voxento 4.0 – an interactive voice-based retrieval system for lifelogs which has been developed to participate in the sixth Lifelog Search Challenge LSC’23, at ACM ICMR’23. Voxento has participated three times in the LSC editions and achieved the rank of 4th in LSC21 and 5th in LSC22 respectively. In this version, Voxento 4.0, we have focused on improving the previous system’s interface, voice interaction and retrieval functionality. The current version has implemented some processing and cleaning of the dataset and employs the CLIP model to extract image features. In addition, the system’s interface was redesigned for better visualisation of the elements and the images for effective interaction. This improvement in the interface will help to support voice interaction in future work. The interface developments include logging voice interaction and images displayed, submitted, selected and starred to enhance user experience with the system. The voice interaction part has also been enhanced in the workflow of the voice lifecycle interaction and with additional voice commands.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"119 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124601491","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ricardo F. Ribeiro, Luísa Amaral, Wei Ye, A. Trifan, António J. R. Neves, Pedro Iglésias
The continuous collection and storage of personal data, denoted Lifelogging, has gained popularity in recent years as a means of monitoring and improving personal health. One important aspect of lifelogging is the collection and analysis of image data, which can provide valuable insights into an individual’s lifestyle, dietary habits, and physical activity. The Lifelog Search Challenge provides a unique opportunity to explore the state-of-the-art in lifelogging research, particularly in the area of egocentric image retrieval and analysis. Researchers can propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems on a rich multimodal dataset generated by an active lifelogger with 18 months of continuous capture of lifelogging data. This paper presents the second version of MEMORIA, a computational tool developed to participate in the Lifelog Search Challenge 2023. In this new version, the information retrieval is based on the use of natural language search with the possibility to filter the results based on keywords and time periods. The system applies image analysis algorithms to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. This new version explores the use of a graph database, more detailed image annotation, and event segmentation, in order to improve the performance and user interaction. Experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.
{"title":"MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2023","authors":"Ricardo F. Ribeiro, Luísa Amaral, Wei Ye, A. Trifan, António J. R. Neves, Pedro Iglésias","doi":"10.1145/3592573.3593099","DOIUrl":"https://doi.org/10.1145/3592573.3593099","url":null,"abstract":"The continuous collection and storage of personal data, denoted Lifelogging, has gained popularity in recent years as a means of monitoring and improving personal health. One important aspect of lifelogging is the collection and analysis of image data, which can provide valuable insights into an individual’s lifestyle, dietary habits, and physical activity. The Lifelog Search Challenge provides a unique opportunity to explore the state-of-the-art in lifelogging research, particularly in the area of egocentric image retrieval and analysis. Researchers can propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems on a rich multimodal dataset generated by an active lifelogger with 18 months of continuous capture of lifelogging data. This paper presents the second version of MEMORIA, a computational tool developed to participate in the Lifelog Search Challenge 2023. In this new version, the information retrieval is based on the use of natural language search with the possibility to filter the results based on keywords and time periods. The system applies image analysis algorithms to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. This new version explores the use of a graph database, more detailed image annotation, and event segmentation, in order to improve the performance and user interaction. Experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114227462","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Florian Spiess, Ralph Gasser, H. Schuldt, Luca Rossetto
Personal lifelog data collections are becoming more common as a memory aid, as well as for analytical tasks, such as health and fitness analysis. Due to the multimodal and personal nature of lifelog data, interactive multimedia retrieval approaches are required to facilitate flexible and iterative query formulation and result exploration for retrieval and analysis. In recent years, novel user interface modalities have emerged, that allow new ways for users to interact with a retrieval system. Virtual reality, one such new modality, provides advantages as well as challenges for interactive multimedia retrieval in comparison to conventional desktop-based interfaces. This paper describes a novel desktop-virtual reality hybrid system participating in the Lifelog Search Challenge 2023. The system, which is based on the components of the vitrivr stack, is described with a focus on query formulation in the web-based desktop user interface vitrivr-ng, and result exploration in the virtual reality-based vitrivr-VR.
{"title":"The Best of Both Worlds: Lifelog Retrieval with a Desktop-Virtual Reality Hybrid System","authors":"Florian Spiess, Ralph Gasser, H. Schuldt, Luca Rossetto","doi":"10.1145/3592573.3593107","DOIUrl":"https://doi.org/10.1145/3592573.3593107","url":null,"abstract":"Personal lifelog data collections are becoming more common as a memory aid, as well as for analytical tasks, such as health and fitness analysis. Due to the multimodal and personal nature of lifelog data, interactive multimedia retrieval approaches are required to facilitate flexible and iterative query formulation and result exploration for retrieval and analysis. In recent years, novel user interface modalities have emerged, that allow new ways for users to interact with a retrieval system. Virtual reality, one such new modality, provides advantages as well as challenges for interactive multimedia retrieval in comparison to conventional desktop-based interfaces. This paper describes a novel desktop-virtual reality hybrid system participating in the Lifelog Search Challenge 2023. The system, which is based on the components of the vitrivr stack, is described with a focus on query formulation in the web-based desktop user interface vitrivr-ng, and result exploration in the virtual reality-based vitrivr-VR.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114864668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this work, we present our system Memento 3.0 for participation in the Lifelog Search Challenge 2023, which is a successor to the previous 2 iterations of our system called Memento 1.0 [1] and Memento 2.0 [2]. Memento 3.0 employs image-text embeddings derived from OpenAI CLIP models as well as larger OpenCLIP models trained on ∼ 5x more data. Our system also significantly reduces the query processing time by almost 75% when compared to its predecessor systems by employing a cluster-based search technique. We additionally make important updates to the system’s user interface to offer more flexibility to the user and at the same time be better suited to efficiently handle new query types introduced in the Lifelog Search Challenge.
{"title":"Memento 3.0: An Enhanced Lifelog Search Engine for LSC’23","authors":"Naushad Alam, Yvette Graham, C. Gurrin","doi":"10.1145/3592573.3593103","DOIUrl":"https://doi.org/10.1145/3592573.3593103","url":null,"abstract":"In this work, we present our system Memento 3.0 for participation in the Lifelog Search Challenge 2023, which is a successor to the previous 2 iterations of our system called Memento 1.0 [1] and Memento 2.0 [2]. Memento 3.0 employs image-text embeddings derived from OpenAI CLIP models as well as larger OpenCLIP models trained on ∼ 5x more data. Our system also significantly reduces the query processing time by almost 75% when compared to its predecessor systems by employing a cluster-based search technique. We additionally make important updates to the system’s user interface to offer more flexibility to the user and at the same time be better suited to efficiently handle new query types introduced in the Lifelog Search Challenge.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124082907","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Searching substantial data archives of lifeloggers is a challenging task. The Lifelog Search Challenge (LSC) is an annually held competition with the aim of encouraging international teams to develop interactive content retrieval systems capable of searching large lifelog databases. LSC takes place as a live event co-located with the ACM International Conference on Multimedia Retrieval (ICMR), where teams compete against each other by solving retrieval tasks issued by the lifelogger. This paper presents our newest version of lifeXplore, a lifelog retrieval system that has been participating in LSC since 2018. For this year, we significantly redesign the entire system (backend, middleware, and frontend) and integrate free text-search using embeddings from vision transformers trained with large sets of text-image pairs. We present a novel architecture for multi-source search, where results from image embeddings are used together with results from traditional content analysis (for objects, concepts, and recognized text). We also perform intensive analysis of vision transformer models in order to know which one fits best to the requirements of the LSC.
{"title":"lifeXplore at the Lifelog Search Challenge 2023","authors":"Klaus Schoeffmann","doi":"10.1145/3592573.3593105","DOIUrl":"https://doi.org/10.1145/3592573.3593105","url":null,"abstract":"Searching substantial data archives of lifeloggers is a challenging task. The Lifelog Search Challenge (LSC) is an annually held competition with the aim of encouraging international teams to develop interactive content retrieval systems capable of searching large lifelog databases. LSC takes place as a live event co-located with the ACM International Conference on Multimedia Retrieval (ICMR), where teams compete against each other by solving retrieval tasks issued by the lifelogger. This paper presents our newest version of lifeXplore, a lifelog retrieval system that has been participating in LSC since 2018. For this year, we significantly redesign the entire system (backend, middleware, and frontend) and integrate free text-search using embeddings from vision transformers trained with large sets of text-image pairs. We present a novel architecture for multi-source search, where results from image embeddings are used together with results from traditional content analysis (for objects, concepts, and recognized text). We also perform intensive analysis of vision transformer models in order to know which one fits best to the requirements of the LSC.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124930471","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Luca Rossetto, O. Inel, Svenja Lange, Florian Ruosch, Ruijie Wang, Abraham Bernstein
As part of the 6th Lifelog Search Challenge, this paper presents an approach to arrange Lifelog data in a multi-modal knowledge graph based on cluster hierarchies. We use multiple sequence clustering approaches to address the multi-modal nature of Lifelogs in relation to temporal, spatial, and visual factors. The resulting clusters, along with semantic metadata captions and augmentations based on OpenCLIP, provide for the semantic structure of a graph including all Lifelogs as entries. Textual queries on this hierarchical graph can be expressed to retrieve individual Lifelogs, as well as clusters of Lifelogs.
{"title":"Multi-Mode Clustering for Graph-Based Lifelog Retrieval","authors":"Luca Rossetto, O. Inel, Svenja Lange, Florian Ruosch, Ruijie Wang, Abraham Bernstein","doi":"10.1145/3592573.3593102","DOIUrl":"https://doi.org/10.1145/3592573.3593102","url":null,"abstract":"As part of the 6th Lifelog Search Challenge, this paper presents an approach to arrange Lifelog data in a multi-modal knowledge graph based on cluster hierarchies. We use multiple sequence clustering approaches to address the multi-modal nature of Lifelogs in relation to temporal, spatial, and visual factors. The resulting clusters, along with semantic metadata captions and augmentations based on OpenCLIP, provide for the semantic structure of a graph including all Lifelogs as entries. Textual queries on this hierarchical graph can be expressed to retrieve individual Lifelogs, as well as clusters of Lifelogs.","PeriodicalId":147486,"journal":{"name":"Proceedings of the 6th Annual ACM Lifelog Search Challenge","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125970453","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：