Yizhu Jiao, Ming Zhong, Jiaming Shen, Yunyi Zhang, Chao Zhang, Jiawei Han
Massive and fast-evolving news articles keep emerging on the web. To effectively summarize and provide concise insights into real-world events, we propose a new event knowledge extraction task Event Chain Mining in this paper. Given multiple documents about a super event, it aims to mine a series of salient events in temporal order. For example, the event chain of super event Mexico Earthquake in 2017 is {earthquake hit Mexico, destroy houses, kill people, block roads}. This task can help readers capture the gist of texts quickly, thereby improving reading efficiency and deepening text comprehension. To address this task, we regard an event as a cluster of different mentions of similar meanings. In this way, we can identify the different expressions of events, enrich their semantic knowledge and replenish relation information among them. Taking events as the basic unit, we present a novel unsupervised framework, EMiner. Specifically, we extract event mentions from texts and merge them with similar meanings into a cluster as a single event. By jointly incorporating both content and commonsense, essential events are then selected and arranged chronologically to form an event chain. Meanwhile, we annotate a multi-document benchmark to build a comprehensive testbed for the proposed task. Extensive experiments are conducted to verify the effectiveness of EMiner in terms of both automatic and human evaluations.
{"title":"Unsupervised Event Chain Mining from Multiple Documents","authors":"Yizhu Jiao, Ming Zhong, Jiaming Shen, Yunyi Zhang, Chao Zhang, Jiawei Han","doi":"10.1145/3543507.3583295","DOIUrl":"https://doi.org/10.1145/3543507.3583295","url":null,"abstract":"Massive and fast-evolving news articles keep emerging on the web. To effectively summarize and provide concise insights into real-world events, we propose a new event knowledge extraction task Event Chain Mining in this paper. Given multiple documents about a super event, it aims to mine a series of salient events in temporal order. For example, the event chain of super event Mexico Earthquake in 2017 is {earthquake hit Mexico, destroy houses, kill people, block roads}. This task can help readers capture the gist of texts quickly, thereby improving reading efficiency and deepening text comprehension. To address this task, we regard an event as a cluster of different mentions of similar meanings. In this way, we can identify the different expressions of events, enrich their semantic knowledge and replenish relation information among them. Taking events as the basic unit, we present a novel unsupervised framework, EMiner. Specifically, we extract event mentions from texts and merge them with similar meanings into a cluster as a single event. By jointly incorporating both content and commonsense, essential events are then selected and arranged chronologically to form an event chain. Meanwhile, we annotate a multi-document benchmark to build a comprehensive testbed for the proposed task. Extensive experiments are conducted to verify the effectiveness of EMiner in terms of both automatic and human evaluations.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131495529","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Christina Yeung, U. Iqbal, Y. O'Neil, Tadayoshi Kohno, Franziska Roesner
Online ads are a major source of information on the web. The mass reach of online advertising is often leveraged for information dissemination, at times with an objective to influence public opinion (e.g., election misinformation). We hypothesized that online advertising, due to its reach and potential, might have been used to spread information around the 2022 Russian invasion of Ukraine. Thus, to understand the online ad ecosystem during this conflict, we conducted a five-month long large-scale measurement study of online advertising in Ukraine, Russia, and the US. We studied advertising trends of ad platforms that delivered ads in Ukraine, Russia, and the US and conducted an in-depth qualitative analysis of the conflict-related ad content. We found that prominent US-based advertisers continued to support Russian websites, and a portion of online ads were used to spread conflict-related information, including protesting the invasion, and spreading awareness, which might have otherwise potentially been censored in Russia.
{"title":"Online Advertising in Ukraine and Russia During the 2022 Russian Invasion","authors":"Christina Yeung, U. Iqbal, Y. O'Neil, Tadayoshi Kohno, Franziska Roesner","doi":"10.1145/3543507.3583484","DOIUrl":"https://doi.org/10.1145/3543507.3583484","url":null,"abstract":"Online ads are a major source of information on the web. The mass reach of online advertising is often leveraged for information dissemination, at times with an objective to influence public opinion (e.g., election misinformation). We hypothesized that online advertising, due to its reach and potential, might have been used to spread information around the 2022 Russian invasion of Ukraine. Thus, to understand the online ad ecosystem during this conflict, we conducted a five-month long large-scale measurement study of online advertising in Ukraine, Russia, and the US. We studied advertising trends of ad platforms that delivered ads in Ukraine, Russia, and the US and conducted an in-depth qualitative analysis of the conflict-related ad content. We found that prominent US-based advertisers continued to support Russian websites, and a portion of online ads were used to spread conflict-related information, including protesting the invasion, and spreading awareness, which might have otherwise potentially been censored in Russia.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"50 11","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120915905","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The repeated user-item interaction now is becoming a common phenomenon in the e-commerce scenario. Due to its potential economic profit, various models are emerging to predict which item will be re-interacted based on the user-item interactions. In this specific scenario, item relevance is a critical factor that needs to be concerned, which tends to have different effects on the succeeding re-interacted one (i.e., stimulating or delaying its emergence). It is necessary to make a detailed discernment of item relevance for a better repetition-aware recommendation. Unfortunately, existing works usually mixed all these types, which may disturb the learning process and result in poor performance. In this paper, we introduce a novel Communicative MARL-based Relevance Discerning Network (CARDfor short) to automatically discern the item relevance for a better repetition-aware recommendation. Specifically, CARDformalizes the item relevance discerning problem into a communication selection process in MARL. CARDtreats each unique interacted item as an agent and defines three different communication types over agents, which are stimulative, inhibitive, and noisy respectively. After this, CARDutilizes a Gumbel-enhanced classifier to distinguish the communication types among agents, and an attention-based Reactive Point Process is further designed to transmit the well-discerned stimulative and inhibitive incentives separately among all agents to make an effective collaboration for repetition decisions. Experimental results on two real-world e-commerce datasets show that our proposed method outperforms the state-of-the-art recommendation methods in terms of both sequential and repetition-aware recommenders. Furthermore, CARDis also deployed in the online sponsored search advertising system in Meituan, obtaining a performance improvement of over 1.5% and 1.2% in CTR and effective Cost Per Mille (eCPM) respectively, which is significant to the business.
在电子商务场景中，重复的用户-物品交互正在成为一种普遍现象。由于其潜在的经济利润，各种各样的模型正在出现，以预测哪些物品将基于用户-物品交互而重新交互。在这个特定的场景中，项目相关性是一个需要关注的关键因素，它往往会对后续的重新交互产生不同的影响(即刺激或延迟其出现)。有必要对项目相关性进行详细的识别，以便更好地提供有重复意识的建议。不幸的是，现有的作品通常混合了所有这些类型，这可能会干扰学习过程，导致表现不佳。在本文中，我们引入了一种新的基于交际marl的关联识别网络(简称card)来自动识别项目相关性，以便更好地进行重复感知推荐。具体而言，cardd将项目相关性识别问题形式化为MARL中的通信选择过程。cardcard将每个唯一的交互项目视为一个代理，并定义了代理上三种不同的通信类型，分别是刺激型、抑制性和噪声型。在此基础上，利用gumbel增强分类器区分智能体之间的通信类型，并进一步设计了基于注意力的反应点过程(Reactive Point Process)，在所有智能体之间分别传递识别好的激励和抑制激励，从而有效地协作进行重复决策。在两个真实电子商务数据集上的实验结果表明，我们提出的方法在顺序和重复感知推荐方面都优于最先进的推荐方法。此外，CARDis还部署在美团的在线赞助搜索广告系统中，在CTR和有效每英里成本(eCPM)方面分别获得了超过1.5%和1.2%的性能提升，这对业务具有重要意义。
{"title":"Communicative MARL-based Relevance Discerning Network for Repetition-Aware Recommendation","authors":"Kaiyuan Li, Pengfei Wang, Haitao Wang, Q. Liu, Xingxing Wang, Dong Wang, Shangguang Wang","doi":"10.1145/3543507.3583459","DOIUrl":"https://doi.org/10.1145/3543507.3583459","url":null,"abstract":"The repeated user-item interaction now is becoming a common phenomenon in the e-commerce scenario. Due to its potential economic profit, various models are emerging to predict which item will be re-interacted based on the user-item interactions. In this specific scenario, item relevance is a critical factor that needs to be concerned, which tends to have different effects on the succeeding re-interacted one (i.e., stimulating or delaying its emergence). It is necessary to make a detailed discernment of item relevance for a better repetition-aware recommendation. Unfortunately, existing works usually mixed all these types, which may disturb the learning process and result in poor performance. In this paper, we introduce a novel Communicative MARL-based Relevance Discerning Network (CARDfor short) to automatically discern the item relevance for a better repetition-aware recommendation. Specifically, CARDformalizes the item relevance discerning problem into a communication selection process in MARL. CARDtreats each unique interacted item as an agent and defines three different communication types over agents, which are stimulative, inhibitive, and noisy respectively. After this, CARDutilizes a Gumbel-enhanced classifier to distinguish the communication types among agents, and an attention-based Reactive Point Process is further designed to transmit the well-discerned stimulative and inhibitive incentives separately among all agents to make an effective collaboration for repetition decisions. Experimental results on two real-world e-commerce datasets show that our proposed method outperforms the state-of-the-art recommendation methods in terms of both sequential and repetition-aware recommenders. Furthermore, CARDis also deployed in the online sponsored search advertising system in Meituan, obtaining a performance improvement of over 1.5% and 1.2% in CTR and effective Cost Per Mille (eCPM) respectively, which is significant to the business.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121383442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hui Luo, Z. Bao, J. Culpepper, Mingzhao Li, Yanchang Zhao
In this paper, we propose a novel facility relocation problem where facilities (and their services) are portable, which is a combinatorial search problem with many practical applications. Given a set of users, a set of existing facilities, and a set of potential sites, we decide which of the existing facilities to relocate to potential sites, such that two factors are satisfied: (1) facility exposure: facilities after relocation have balanced exposure, namely serving equivalent numbers of users; (2) user convenience: it is convenient for users to access the nearest facility, which provides services with shorter travel distance. This problem is motivated by applications such as dynamically redistributing vaccine resources to align supply with demand for different vaccination centers, and relocating the bike sharing sites daily to improve the transportation efficiency. We first prove that this problem is NP-hard, and then we propose two algorithms: a non-learning best response algorithm () and a reinforcement learning algorithm (). In particular, the best response algorithm finds a Nash equilibrium to balance the facility-related and the user-related goals. To avoid being confined to only one Nash equilibrium, as found in the method, we also propose the reinforcement learning algorithm for long-term benefits, where each facility is an agent and we determine whether a facility needs to be relocated or not. To verify the effectiveness of our methods, we adopt multiple metrics to evaluate not only our objective, but also several other facility exposure equity and user convenience metrics to understand the benefits after facility relocation. Finally, comprehensive experiments using real-world datasets provide insights into the effectiveness of the two algorithms in practice.
{"title":"Facility Relocation Search For Good: When Facility Exposure Meets User Convenience","authors":"Hui Luo, Z. Bao, J. Culpepper, Mingzhao Li, Yanchang Zhao","doi":"10.1145/3543507.3583859","DOIUrl":"https://doi.org/10.1145/3543507.3583859","url":null,"abstract":"In this paper, we propose a novel facility relocation problem where facilities (and their services) are portable, which is a combinatorial search problem with many practical applications. Given a set of users, a set of existing facilities, and a set of potential sites, we decide which of the existing facilities to relocate to potential sites, such that two factors are satisfied: (1) facility exposure: facilities after relocation have balanced exposure, namely serving equivalent numbers of users; (2) user convenience: it is convenient for users to access the nearest facility, which provides services with shorter travel distance. This problem is motivated by applications such as dynamically redistributing vaccine resources to align supply with demand for different vaccination centers, and relocating the bike sharing sites daily to improve the transportation efficiency. We first prove that this problem is NP-hard, and then we propose two algorithms: a non-learning best response algorithm () and a reinforcement learning algorithm (). In particular, the best response algorithm finds a Nash equilibrium to balance the facility-related and the user-related goals. To avoid being confined to only one Nash equilibrium, as found in the method, we also propose the reinforcement learning algorithm for long-term benefits, where each facility is an agent and we determine whether a facility needs to be relocated or not. To verify the effectiveness of our methods, we adopt multiple metrics to evaluate not only our objective, but also several other facility exposure equity and user convenience metrics to understand the benefits after facility relocation. Finally, comprehensive experiments using real-world datasets provide insights into the effectiveness of the two algorithms in practice.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"235 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116287856","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Mengqi Zhang, Yuwei Xia, Q. Liu, Shu Wu, Liang Wang
Temporal Knowledge graph (TKG) reasoning aims to predict missing facts based on historical TKG data. Most of the existing methods are incapable of explicitly modeling the long-term time dependencies from history and neglect the adaptive integration of the long- and short-term information. To tackle these problems, we propose a novel method that utilizes a designed Hierarchical Relational Graph Neural Network to learn the Long- and Short-term representations for TKG reasoning, namely HGLS. Specifically, to explicitly associate entities in different timestamps, we first transform the TKG into a global graph. Based on the built graph, we design a Hierarchical Relational Graph Neural Network that executes in two levels: The sub-graph level is to capture the semantic dependencies within concurrent facts of each KG. And the global-graph level aims to model the temporal dependencies between entities. Furthermore, we design a module to extract the long- and short-term information from the output of these two levels. Finally, the long- and short-term representations are fused into a unified one by Gating Integration for entity prediction. Extensive experiments on four datasets demonstrate the effectiveness of HGLS.
{"title":"Learning Long- and Short-term Representations for Temporal Knowledge Graph Reasoning","authors":"Mengqi Zhang, Yuwei Xia, Q. Liu, Shu Wu, Liang Wang","doi":"10.1145/3543507.3583242","DOIUrl":"https://doi.org/10.1145/3543507.3583242","url":null,"abstract":"Temporal Knowledge graph (TKG) reasoning aims to predict missing facts based on historical TKG data. Most of the existing methods are incapable of explicitly modeling the long-term time dependencies from history and neglect the adaptive integration of the long- and short-term information. To tackle these problems, we propose a novel method that utilizes a designed Hierarchical Relational Graph Neural Network to learn the Long- and Short-term representations for TKG reasoning, namely HGLS. Specifically, to explicitly associate entities in different timestamps, we first transform the TKG into a global graph. Based on the built graph, we design a Hierarchical Relational Graph Neural Network that executes in two levels: The sub-graph level is to capture the semantic dependencies within concurrent facts of each KG. And the global-graph level aims to model the temporal dependencies between entities. Furthermore, we design a module to extract the long- and short-term information from the output of these two levels. Finally, the long- and short-term representations are fused into a unified one by Gating Integration for entity prediction. Extensive experiments on four datasets demonstrate the effectiveness of HGLS.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126905425","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Weiming Liu, Xiaolin Zheng, Chaochao Chen, Jiajie Su, Xinting Liao, Mengling Hu, Yanchao Tan
Sequential Cross-Domain Recommendation (CDR) has been popularly studied to utilize different domain knowledge and users’ historical behaviors for the next-item prediction. In this paper, we focus on the cross-domain sequential recommendation problem. This commonly exist problem is rather challenging from two perspectives, i.e., the implicit user historical rating sequences are difficult in modeling and the users/items on different domains are mostly non-overlapped. Most previous sequential CDR approaches cannot solve the cross-domain sequential recommendation problem well, since (1) they cannot sufficiently depict the users’ actual preferences, (2) they cannot leverage and transfer useful knowledge across domains. To tackle the above issues, we propose joint Internal multi-interest exploration and External domain alignment for cross domain Sequential Recommendation model (IESRec). IESRec includes two main modules, i.e., internal multi-interest exploration module and external domain alignment module. To reflect the users’ diverse characteristics with multi-interests evolution, we first propose internal temporal optimal transport method in the internal multi-interest exploration module. We further propose external alignment optimal transport method in the external domain alignment module to reduce domain discrepancy for the item embeddings. Our empirical studies on Amazon datasets demonstrate that IESRec significantly outperforms the state-of-the-art models.
{"title":"Joint Internal Multi-Interest Exploration and External Domain Alignment for Cross Domain Sequential Recommendation","authors":"Weiming Liu, Xiaolin Zheng, Chaochao Chen, Jiajie Su, Xinting Liao, Mengling Hu, Yanchao Tan","doi":"10.1145/3543507.3583366","DOIUrl":"https://doi.org/10.1145/3543507.3583366","url":null,"abstract":"Sequential Cross-Domain Recommendation (CDR) has been popularly studied to utilize different domain knowledge and users’ historical behaviors for the next-item prediction. In this paper, we focus on the cross-domain sequential recommendation problem. This commonly exist problem is rather challenging from two perspectives, i.e., the implicit user historical rating sequences are difficult in modeling and the users/items on different domains are mostly non-overlapped. Most previous sequential CDR approaches cannot solve the cross-domain sequential recommendation problem well, since (1) they cannot sufficiently depict the users’ actual preferences, (2) they cannot leverage and transfer useful knowledge across domains. To tackle the above issues, we propose joint Internal multi-interest exploration and External domain alignment for cross domain Sequential Recommendation model (IESRec). IESRec includes two main modules, i.e., internal multi-interest exploration module and external domain alignment module. To reflect the users’ diverse characteristics with multi-interests evolution, we first propose internal temporal optimal transport method in the internal multi-interest exploration module. We further propose external alignment optimal transport method in the external domain alignment module to reduce domain discrepancy for the item embeddings. Our empirical studies on Amazon datasets demonstrate that IESRec significantly outperforms the state-of-the-art models.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126874583","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Vector quantization techniques, such as Product Quantization (PQ), play a vital role in approximate nearest neighbor search (ANNs) and maximum inner product search (MIPS) owing to their remarkable search and storage efficiency. However, the indexes in vector quantization cannot be trained together with the inference models since data indexing is not differentiable. To this end, differentiable vector quantization approaches, such as DiffPQ and DeepPQ, have been recently proposed, but existing methods have two drawbacks. First, they do not impose any constraints on codebooks, such that the resultant codebooks lack diversity, leading to limited retrieval performance. Second, since data indexing resorts to operator, differentiability is usually achieved by either relaxation or Straight-Through Estimation (STE), which leads to biased gradient and slow convergence. To address these problems, we propose a Differentiable Optimized Product Quantization method (DOPQ) and beyond in this paper. Particularly, each data is projected into multiple orthogonal spaces, to generate multiple views of data. Thus, each codebook is learned with one view of data, guaranteeing the diversity of codebooks. Moreover, instead of simple differentiable relaxation, DOPQ optimizes the loss based on direct loss minimization, significantly reducing the gradient bias problem. Finally, DOPQ is evaluated with seven datasets of both recommendation and image search tasks. Extensive experimental results show that DOPQ outperforms state-of-the-art baselines by a large margin.
{"title":"Differentiable Optimized Product Quantization and Beyond","authors":"Zepu Lu, Defu Lian, Jin Zhang, Zaixin Zhang, Chao Feng, Hao Wang, Enhong Chen","doi":"10.1145/3543507.3583482","DOIUrl":"https://doi.org/10.1145/3543507.3583482","url":null,"abstract":"Vector quantization techniques, such as Product Quantization (PQ), play a vital role in approximate nearest neighbor search (ANNs) and maximum inner product search (MIPS) owing to their remarkable search and storage efficiency. However, the indexes in vector quantization cannot be trained together with the inference models since data indexing is not differentiable. To this end, differentiable vector quantization approaches, such as DiffPQ and DeepPQ, have been recently proposed, but existing methods have two drawbacks. First, they do not impose any constraints on codebooks, such that the resultant codebooks lack diversity, leading to limited retrieval performance. Second, since data indexing resorts to operator, differentiability is usually achieved by either relaxation or Straight-Through Estimation (STE), which leads to biased gradient and slow convergence. To address these problems, we propose a Differentiable Optimized Product Quantization method (DOPQ) and beyond in this paper. Particularly, each data is projected into multiple orthogonal spaces, to generate multiple views of data. Thus, each codebook is learned with one view of data, guaranteeing the diversity of codebooks. Moreover, instead of simple differentiable relaxation, DOPQ optimizes the loss based on direct loss minimization, significantly reducing the gradient bias problem. Finally, DOPQ is evaluated with seven datasets of both recommendation and image search tasks. Extensive experimental results show that DOPQ outperforms state-of-the-art baselines by a large margin.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"58 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126257004","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts’ ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to evaluate a large number of design ideas in a short amount of time while being cost-effective. Workers’ evaluation is, however, less reliable and might substantially differ from experts’ evaluation. In this work, we investigate workers’ rating behavior and compare it with experts. First, we instrument a crowdsourcing study where we asked workers to evaluate design ideas from three innovation challenges. We show that workers share similar insights with experts but tend to rate more generously and weigh certain criteria more importantly. Next, we develop a hybrid human-AI approach that combines a machine learning model with crowdsourcing to evaluate ideas. Our approach models workers’ reliability and bias while leveraging ideas’ textual content to train a machine learning model. It is able to incorporate experts’ ratings whenever available, to supervise the model training and infer worker performance. Results show that our framework outperforms baseline methods and requires significantly less training data from experts, thus providing a viable solution for evaluating ideas at scale.
{"title":"HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale","authors":"S. Mesbah, Ines Arous, Jie Yang, A. Bozzon","doi":"10.1145/3543507.3583496","DOIUrl":"https://doi.org/10.1145/3543507.3583496","url":null,"abstract":"Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts’ ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to evaluate a large number of design ideas in a short amount of time while being cost-effective. Workers’ evaluation is, however, less reliable and might substantially differ from experts’ evaluation. In this work, we investigate workers’ rating behavior and compare it with experts. First, we instrument a crowdsourcing study where we asked workers to evaluate design ideas from three innovation challenges. We show that workers share similar insights with experts but tend to rate more generously and weigh certain criteria more importantly. Next, we develop a hybrid human-AI approach that combines a machine learning model with crowdsourcing to evaluate ideas. Our approach models workers’ reliability and bias while leveraging ideas’ textual content to train a machine learning model. It is able to incorporate experts’ ratings whenever available, to supervise the model training and infer worker performance. Results show that our framework outperforms baseline methods and requires significantly less training data from experts, thus providing a viable solution for evaluating ideas at scale.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126423078","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Linsen Li, A. Culotta, Douglas N. Harris, Nicholas Mattei
School rating websites are increasingly used by parents to assess the quality and fit of U.S. K-12 schools for their children. These online reviews often contain detailed descriptions of a school’s strengths and weaknesses, which both reflect and inform perceptions of a school. Existing work on these text reviews has focused on finding words or themes that underlie these perceptions, but has stopped short of using the textual reviews as leading indicators of school performance. In this paper, we investigate to what extent the language used in online reviews of a school is predictive of changes in the attributes of that school, such as its socio-economic makeup and student test scores. Using over 300K reviews of 70K U.S. schools from a popular ratings website, we apply language processing models to predict whether schools will significantly increase or decrease in an attribute of interest over a future time horizon. We find that using the text improves predictive performance significantly over a baseline model that does not include text but only the historical time-series of the indicators themselves, suggesting that the review text carries predictive power. A qualitative analysis of the most predictive terms and phrases used in the text reviews indicates a number of topics that serve as leading indicators, such as diversity, changes in school leadership, a focus on testing, and school safety.
{"title":"Online Reviews Are Leading Indicators of Changes in K-12 School Attributes","authors":"Linsen Li, A. Culotta, Douglas N. Harris, Nicholas Mattei","doi":"10.1145/3543507.3583531","DOIUrl":"https://doi.org/10.1145/3543507.3583531","url":null,"abstract":"School rating websites are increasingly used by parents to assess the quality and fit of U.S. K-12 schools for their children. These online reviews often contain detailed descriptions of a school’s strengths and weaknesses, which both reflect and inform perceptions of a school. Existing work on these text reviews has focused on finding words or themes that underlie these perceptions, but has stopped short of using the textual reviews as leading indicators of school performance. In this paper, we investigate to what extent the language used in online reviews of a school is predictive of changes in the attributes of that school, such as its socio-economic makeup and student test scores. Using over 300K reviews of 70K U.S. schools from a popular ratings website, we apply language processing models to predict whether schools will significantly increase or decrease in an attribute of interest over a future time horizon. We find that using the text improves predictive performance significantly over a baseline model that does not include text but only the historical time-series of the indicators themselves, suggesting that the review text carries predictive power. A qualitative analysis of the most predictive terms and phrases used in the text reviews indicates a number of topics that serve as leading indicators, such as diversity, changes in school leadership, a focus on testing, and school safety.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133622731","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hsi-Wen Chen, De-Nian Yang, Wang-Chien Lee, P. Yu, Ming-Syan Chen
The phenomena of influence diffusion on social networks have received tremendous research interests in the past decade. While most prior works mainly focus on predicting the total influence spread on a single network, a marketing campaign that exploits influence diffusion often involves multiple channels with various information disseminated on different media. In this paper, we introduce a new influence estimation problem, namely Content-aware Multi-channel Influence Diffusion (CMID), and accordingly propose CMINet to predict newly influenced users, given a set of seed users with different multimedia contents. In CMINet, we first introduce DiffGNN to encode the influencing power of users (nodes) and Influence-aware Optimal Transport (IOT) to align the embeddings to address the distribution shift across different diffusion channels. Then, we transform CMID into a node classification problem and propose Social-based Multimedia Feature Extractor (SMFE) and Content-aware Multi-channel Influence Propagation (CMIP) to jointly learn the user preferences on multimedia contents and predict the susceptibility of users. Furthermore, we prove that CMINet preserves monotonicity and submodularity, thus enabling (1 − 1/e)-approximate solutions for influence maximization. Experimental results manifest that CMINet outperforms eleven baselines on three public datasets.
{"title":"CMINet: a Graph Learning Framework for Content-aware Multi-channel Influence Diffusion","authors":"Hsi-Wen Chen, De-Nian Yang, Wang-Chien Lee, P. Yu, Ming-Syan Chen","doi":"10.1145/3543507.3583465","DOIUrl":"https://doi.org/10.1145/3543507.3583465","url":null,"abstract":"The phenomena of influence diffusion on social networks have received tremendous research interests in the past decade. While most prior works mainly focus on predicting the total influence spread on a single network, a marketing campaign that exploits influence diffusion often involves multiple channels with various information disseminated on different media. In this paper, we introduce a new influence estimation problem, namely Content-aware Multi-channel Influence Diffusion (CMID), and accordingly propose CMINet to predict newly influenced users, given a set of seed users with different multimedia contents. In CMINet, we first introduce DiffGNN to encode the influencing power of users (nodes) and Influence-aware Optimal Transport (IOT) to align the embeddings to address the distribution shift across different diffusion channels. Then, we transform CMID into a node classification problem and propose Social-based Multimedia Feature Extractor (SMFE) and Content-aware Multi-channel Influence Propagation (CMIP) to jointly learn the user preferences on multimedia contents and predict the susceptibility of users. Furthermore, we prove that CMINet preserves monotonicity and submodularity, thus enabling (1 − 1/e)-approximate solutions for influence maximization. Experimental results manifest that CMINet outperforms eleven baselines on three public datasets.","PeriodicalId":296351,"journal":{"name":"Proceedings of the ACM Web Conference 2023","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131160680","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：