Pub Date : 2023-11-01DOI: 10.1527/tjsai.38-6_c-n22
Yuya Asazuma, Kazuaki Hanawa, Kentaro Inui
Many high-performance machine learning models in the real world exhibit the black box problem. This issue is widely recognized as needing output reliability and model transparency. XAI (Explainable AI) represents a research field that addresses this issue. Within XAI, feature attribution methods, which clarify the importance of features irrespective of the task or model type, have become a central focus. Evaluating their efficacy based on empirical evidence is essential when proposing new methods. However, extensive debate exists regarding the properties that importance should be possessed, and a consensus on specific evaluation methods remains elusive. Given this context, many existing studies adopt their evaluation techniques, leading to fragmented discussions. This study aims to ”evaluate the evaluation methods,” focusing mainly on the faithfulness metric, deemed especially significant in evaluation criteria. We conducted empirical experiments related to existing evaluation techniques. The experiments approached the topic from two angles: correlation-based comparative evaluations and property verification using random sequences. In the former experiment, we investigated the correlation between faithfulness evaluation tests using numerous models and feature attribution methods. As a result, we found that very few test combinations exhibited high correlation, and many combinations showed low or no correlation. In the latter experiment, we observed that the measured faithfulness varied depending on the model and dataset by using random sequences instead of feature attribution methods to verify the properties of the faithfulness tests.
{"title":"Empirical Analysis of Methods for Evaluating Faithfulness of Explanations by Feature Attribution","authors":"Yuya Asazuma, Kazuaki Hanawa, Kentaro Inui","doi":"10.1527/tjsai.38-6_c-n22","DOIUrl":"https://doi.org/10.1527/tjsai.38-6_c-n22","url":null,"abstract":"Many high-performance machine learning models in the real world exhibit the black box problem. This issue is widely recognized as needing output reliability and model transparency. XAI (Explainable AI) represents a research field that addresses this issue. Within XAI, feature attribution methods, which clarify the importance of features irrespective of the task or model type, have become a central focus. Evaluating their efficacy based on empirical evidence is essential when proposing new methods. However, extensive debate exists regarding the properties that importance should be possessed, and a consensus on specific evaluation methods remains elusive. Given this context, many existing studies adopt their evaluation techniques, leading to fragmented discussions. This study aims to ”evaluate the evaluation methods,” focusing mainly on the faithfulness metric, deemed especially significant in evaluation criteria. We conducted empirical experiments related to existing evaluation techniques. The experiments approached the topic from two angles: correlation-based comparative evaluations and property verification using random sequences. In the former experiment, we investigated the correlation between faithfulness evaluation tests using numerous models and feature attribution methods. As a result, we found that very few test combinations exhibited high correlation, and many combinations showed low or no correlation. In the latter experiment, we observed that the measured faithfulness varied depending on the model and dataset by using random sequences instead of feature attribution methods to verify the properties of the faithfulness tests.","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135161382","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt contrastive languageimage pre-training (CLIP), a visual-language foundation model, for DG problems in image classification. While empirical risk minimization (ERM) greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learning (DPL) as a novel approach for domain inference in the form of conditional prompt generation. DPL achieved a significant accuracy improvement with only training a lightweight prompt generator (a three-layer MLP), whose parameter is of equivalent scale to the classification projector in the previous DG literature. Combining DPL with CLIP provides surprising performance, raising the accuracy of zero-shot CLIP from 73.7% to 79.3% on several standard datasets, namely PACS, VLCS, OfficeHome, and TerraIncognita. We hope the simplicity and success of our approach lead to broader adoption and analysis of foundation models in the domain generalization field.
{"title":"Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domains","authors":"Xin Zhang, Shixiang Shane Gu, Yutaka Matsuo, Yusuke Iwasawa","doi":"10.1527/tjsai.38-6_b-mc2","DOIUrl":"https://doi.org/10.1527/tjsai.38-6_b-mc2","url":null,"abstract":"Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt contrastive languageimage pre-training (CLIP), a visual-language foundation model, for DG problems in image classification. While empirical risk minimization (ERM) greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learning (DPL) as a novel approach for domain inference in the form of conditional prompt generation. DPL achieved a significant accuracy improvement with only training a lightweight prompt generator (a three-layer MLP), whose parameter is of equivalent scale to the classification projector in the previous DG literature. Combining DPL with CLIP provides surprising performance, raising the accuracy of zero-shot CLIP from 73.7% to 79.3% on several standard datasets, namely PACS, VLCS, OfficeHome, and TerraIncognita. We hope the simplicity and success of our approach lead to broader adoption and analysis of foundation models in the domain generalization field.","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135161241","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-01DOI: 10.1527/tjsai.38-5_e-n34
Mitsuaki Ueno, Takashi Sano, H. Honda, Shugo Nakamura
{"title":"Detecting Fraudulent Cryptocurrencies Using Natural Language Processing Techniques","authors":"Mitsuaki Ueno, Takashi Sano, H. Honda, Shugo Nakamura","doi":"10.1527/tjsai.38-5_e-n34","DOIUrl":"https://doi.org/10.1527/tjsai.38-5_e-n34","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42679967","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-01DOI: 10.1527/tjsai.38-5_b-mb2
Kazuma Sasaki, Shinya Kitaoka, Yuri Odagiri
{"title":"Diverse Pose Generation for Multi-Joint Characters Conditional on Body Position using IMLE","authors":"Kazuma Sasaki, Shinya Kitaoka, Yuri Odagiri","doi":"10.1527/tjsai.38-5_b-mb2","DOIUrl":"https://doi.org/10.1527/tjsai.38-5_b-mb2","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"67113353","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-01DOI: 10.1527/tjsai.38-5_a-mb1
Shota Imai, Yusuke Iwasawa, Yutaka Matsuo
{"title":"Model-Based Reinforcement Learning using Model Mediator in Dynamic Multi-Agent Environment","authors":"Shota Imai, Yusuke Iwasawa, Yutaka Matsuo","doi":"10.1527/tjsai.38-5_a-mb1","DOIUrl":"https://doi.org/10.1527/tjsai.38-5_a-mb1","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45816069","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-01DOI: 10.1527/tjsai.38-5_d-mc3
Y. Norimatsu
{"title":"Estimating Counterfactual Outcomes of Time-varying Treatments using Deep Gaussian Process","authors":"Y. Norimatsu","doi":"10.1527/tjsai.38-5_d-mc3","DOIUrl":"https://doi.org/10.1527/tjsai.38-5_d-mc3","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45216499","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-07-01DOI: 10.1527/tjsai.38-4_c-mb5
Xing Yan, Yasuharu Den
{"title":"A Proposal of Response Continuity Prediction Model for Attentive Listening Agents","authors":"Xing Yan, Yasuharu Den","doi":"10.1527/tjsai.38-4_c-mb5","DOIUrl":"https://doi.org/10.1527/tjsai.38-4_c-mb5","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46026491","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"日本語文法誤り訂正における事前学習済みモデルを用いたデータ増強","authors":"Hideyoshi Kato, Masaaki Okabe, Michiharu Kitano, Hiroshi Yadohisa","doi":"10.1527/tjsai.38-4_a-l41","DOIUrl":"https://doi.org/10.1527/tjsai.38-4_a-l41","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45397228","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-07-01DOI: 10.1527/tjsai.38-4_b-n11
T. Nagura, E. Akiyama
{"title":"The Effect of Increasing Number of Topics to Polarization and Echo Chambers on SNS","authors":"T. Nagura, E. Akiyama","doi":"10.1527/tjsai.38-4_b-n11","DOIUrl":"https://doi.org/10.1527/tjsai.38-4_b-n11","url":null,"abstract":"","PeriodicalId":23256,"journal":{"name":"Transactions of The Japanese Society for Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47658059","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：