Automatic text summarization techniques have a very high applicability in the legal domain, due to the complex and lengthy nature of legal documents. Most of the classical text summarization algorithms, which are also used in the legal domain, have certain hyperparameters, which if optimized properly, can further improve these algorithms. The choices of these hyperparameters have a big effect on the performance of such algorithms, yet this step of hyperparameter tuning is often overlooked while applying these algorithms in practice. In this work, a Bayesian Optimization based approach is proposed to optimize one of the classical summarization algorithms, Textrank, over this space of choices, by optimizing a ROUGE score mixture based objective function. The process of fine tuning and further evaluation is performed with the help of a publicly available dataset. From the experimental evaluation, it has been observed that the hyperparameter tuned Textrank is able to outperform baseline one-hot vector based Textrank and word2vec based Textrank models, with respect to ROUGE-1, ROUGE-2 and ROUGE-L metrics. The experimental analysis suggests that if proper hyperparameter tuning is performed, even a simple algorithm like Textrank can also perform significantly in the legal document summarization task.
{"title":"Fine-Tuning Textrank for Legal Document Summarization: A Bayesian Optimization Based Approach","authors":"Deepali Jain, M. Borah, A. Biswas","doi":"10.1145/3441501.3441502","DOIUrl":"https://doi.org/10.1145/3441501.3441502","url":null,"abstract":"Automatic text summarization techniques have a very high applicability in the legal domain, due to the complex and lengthy nature of legal documents. Most of the classical text summarization algorithms, which are also used in the legal domain, have certain hyperparameters, which if optimized properly, can further improve these algorithms. The choices of these hyperparameters have a big effect on the performance of such algorithms, yet this step of hyperparameter tuning is often overlooked while applying these algorithms in practice. In this work, a Bayesian Optimization based approach is proposed to optimize one of the classical summarization algorithms, Textrank, over this space of choices, by optimizing a ROUGE score mixture based objective function. The process of fine tuning and further evaluation is performed with the help of a publicly available dataset. From the experimental evaluation, it has been observed that the hyperparameter tuned Textrank is able to outperform baseline one-hot vector based Textrank and word2vec based Textrank models, with respect to ROUGE-1, ROUGE-2 and ROUGE-L metrics. The experimental analysis suggests that if proper hyperparameter tuning is performed, even a simple algorithm like Textrank can also perform significantly in the legal document summarization task.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127888664","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Thomas Mandl, Sandip J Modha, M. Anandkumar, Bharathi Raja Chakravarthi
This paper presents the HASOC track and its two parts. HASOC is dedicated to evaluate technology for finding Offensive Language and Hate Speech. HASOC is creating test collections for languages with few resources and English for comparison. The first track within HASOC has continued work from 2019 and provided a testbed of Twitter posts for Hindi, German and English. The second track within HASOC has created test resources for Tamil and Malayalam in native and Latin script. Posts were extracted mainly from Youtube and Twitter. Both tracks have attracted much interest and over 40 research groups have participated as well as described their approaches in papers. In this overview, we present the tasks, the data and the main results.
{"title":"Overview of the HASOC Track at FIRE 2020: Hate Speech and Offensive Language Identification in Tamil, Malayalam, Hindi, English and German","authors":"Thomas Mandl, Sandip J Modha, M. Anandkumar, Bharathi Raja Chakravarthi","doi":"10.1145/3441501.3441517","DOIUrl":"https://doi.org/10.1145/3441501.3441517","url":null,"abstract":"This paper presents the HASOC track and its two parts. HASOC is dedicated to evaluate technology for finding Offensive Language and Hate Speech. HASOC is creating test collections for languages with few resources and English for comparison. The first track within HASOC has continued work from 2019 and provided a testbed of Twitter posts for Hindi, German and English. The second track within HASOC has created test resources for Tamil and Malayalam in native and Latin script. Posts were extracted mainly from Youtube and Twitter. Both tracks have attracted much interest and over 40 research groups have participated as well as described their approaches in papers. In this overview, we present the tasks, the data and the main results.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"520 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134432112","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Data Fusion combines document rankings from multiple systems into one, in order to improve retrieval effectiveness. Many approaches to this task have been proposed in the literature, and these have been evaluated in various ways. This paper examines a number of such evaluations, to extract commonalities between approaches. Some drawbacks of the prevailing evaluation strategies are then identified, and suggestions made for more appropriate evaluation of data fusion.
{"title":"On the Evaluation of Data Fusion for Information Retrieval","authors":"David Lillis","doi":"10.1145/3441501.3441506","DOIUrl":"https://doi.org/10.1145/3441501.3441506","url":null,"abstract":"Data Fusion combines document rankings from multiple systems into one, in order to improve retrieval effectiveness. Many approaches to this task have been proposed in the literature, and these have been evaluated in various ways. This paper examines a number of such evaluations, to extract commonalities between approaches. Some drawbacks of the prevailing evaluation strategies are then identified, and suggestions made for more appropriate evaluation of data fusion.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133473265","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Fadel, Husam Musleh, Ibraheem Tuffaha, M. Al-Ayyoub, Y. Jararweh, E. Benkhelifa, Paolo Rosso
Authorship identification is essential to the detection of undesirable deception of others’ content misuse or exposing the owners of some anonymous malicious content. While it is widely studied for natural languages, it is rarely considered for programming languages. Accordingly, a PAN@FIRE task, named Authorship Identification of SOurce COde (AI-SOCO), is proposed with the focus on the identification of source code authors. The dataset consists of crawled source codes submitted by the top 1,000 human users with 100 correct C++ submissions or more from the CodeForces online judge platform. The participating systems are asked to predict the author of a given source code from the predefined list of code authors. In total, 60 teams registered on the task’s CodaLab page. Out of them, 14 teams submitted 94 runs. The results are surprisingly high with many teams and baselines breaking the 90% accuracy barrier. These systems used a wide range of models and techniques from pretrained word embeddings (especially, those that are tweaked to handle source code) to stylometric features.
{"title":"Overview of the PAN@FIRE 2020 Task on the Authorship Identification of SOurce COde","authors":"A. Fadel, Husam Musleh, Ibraheem Tuffaha, M. Al-Ayyoub, Y. Jararweh, E. Benkhelifa, Paolo Rosso","doi":"10.1145/3441501.3441532","DOIUrl":"https://doi.org/10.1145/3441501.3441532","url":null,"abstract":"Authorship identification is essential to the detection of undesirable deception of others’ content misuse or exposing the owners of some anonymous malicious content. While it is widely studied for natural languages, it is rarely considered for programming languages. Accordingly, a PAN@FIRE task, named Authorship Identification of SOurce COde (AI-SOCO), is proposed with the focus on the identification of source code authors. The dataset consists of crawled source codes submitted by the top 1,000 human users with 100 correct C++ submissions or more from the CodeForces online judge platform. The participating systems are asked to predict the author of a given source code from the predefined list of code authors. In total, 60 teams registered on the task’s CodaLab page. Out of them, 14 teams submitted 94 runs. The results are surprisingly high with many teams and baselines breaking the 90% accuracy barrier. These systems used a wide range of models and techniques from pretrained word embeddings (especially, those that are tweaked to handle source code) to stylometric features.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115378777","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The FIRE 2020 AILA track aimed at developing datasets and frameworks for the following two tasks: (i) Precedent and Statute Retrieval, where the task was to identify relevant prior cases and statutes (written laws) given a factual scenario, and (ii) Rhetorical Role Labelling for legal judgements, where given a case document, sentences were to be classified into 7 rhetorical roles – Fact, Ruling by Lower Court, Argument, Precedent, Statute, Ratio of the decision and Ruling by Present Court. For both the tasks, we used publicly available Indian Supreme Court case documents.
FIRE 2020 AILA旨在为以下两项任务开发数据集和框架:(i)先例和法规检索，其任务是在给定事实场景的情况下识别相关的先前案例和法规(成文法)，以及(ii)法律判决的修辞角色标签，在给定案件文件时，将句子分为7个修辞角色-事实，下级法院裁决，论证，先例，法规，判决和本院裁决的比例。对于这两项任务，我们都使用了公开的印度最高法院案件文件。
{"title":"FIRE 2020 AILA Track: Artificial Intelligence for Legal Assistance","authors":"Paheli Bhattacharya, Parth Mehta, Kripabandhu Ghosh, Saptarshi Ghosh, Arindam Pal, A. Bhattacharya, Prasenjit Majumder","doi":"10.1145/3441501.3441510","DOIUrl":"https://doi.org/10.1145/3441501.3441510","url":null,"abstract":"The FIRE 2020 AILA track aimed at developing datasets and frameworks for the following two tasks: (i) Precedent and Statute Retrieval, where the task was to identify relevant prior cases and statutes (written laws) given a factual scenario, and (ii) Rhetorical Role Labelling for legal judgements, where given a case document, sentences were to be classified into 7 rhetorical roles – Fact, Ruling by Lower Court, Argument, Precedent, Statute, Ratio of the decision and Ruling by Present Court. For both the tasks, we used publicly available Indian Supreme Court case documents.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131719839","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Resolution of anaphors is required for any application which require Natural Language Understanding (NLU) such as Information Extraction, Conversation Analysis, Opinion Mining, Machine Translation etc. The growth of social media platforms such as twitter, facebook for communication between people has led to the creation of huge user generated data different from the normal text data. This is leading to the development of a new challenge and perspective in language technology research. Thus there is great need to develop applications such as Anaphora resolution, co-reference resolution which can be used for the development of NLU systems. This shared task is on Anaphora resolution from the microblog text from Twitter for languages such as Hindi, Tamil and Malayalam (Indian Languages). Also we gave data from English which can be used as resource rich language, if one wants to take Indian languages as resources poor language. There were six registered groups who took data for development and testing but only one group submitted the run. They have used Deep learning for analysis.
{"title":"Anaphora Resolution from Social Media Text in Indian Languages (SocAnaRes-IL)- Overview","authors":"S. L. Devi","doi":"10.1145/3441501.3441512","DOIUrl":"https://doi.org/10.1145/3441501.3441512","url":null,"abstract":"Resolution of anaphors is required for any application which require Natural Language Understanding (NLU) such as Information Extraction, Conversation Analysis, Opinion Mining, Machine Translation etc. The growth of social media platforms such as twitter, facebook for communication between people has led to the creation of huge user generated data different from the normal text data. This is leading to the development of a new challenge and perspective in language technology research. Thus there is great need to develop applications such as Anaphora resolution, co-reference resolution which can be used for the development of NLU systems. This shared task is on Anaphora resolution from the microblog text from Twitter for languages such as Hindi, Tamil and Malayalam (Indian Languages). Also we gave data from English which can be used as resource rich language, if one wants to take Indian languages as resources poor language. There were six registered groups who took data for development and testing but only one group submitted the run. They have used Deep learning for analysis.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126723783","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present a real world case study for the evaluation of professional search focusing on German construction law. Reliable identification of relevant previous cases is an important part of many legal disputes, and currently relies on domain expertise acquired over a lengthy professional career. We describe our experiences from the development of a Cranfield type test collection for a German construction law dataset to enable research into the development of search technologies for new tools which are less dependent on expert knowledge. We describe examination of the search needs of lawyers, the development of a set of search queries created by lawyers, and our experiences in collecting expert relevance data for the completion of a test collection for legal search. Important findings of this latter process are the need for individuals with expert legal training to assess relevance, and the identification of context dependence in determining relevance. While the cost of the development of this test collection was found to be very high, we demonstrate its value in terms of identifying the effectiveness of legal search methods and in identifying research directions for legal case search.
{"title":"Evaluating Professional Search: A German Construction Law Use Case","authors":"Wei Li, G. Jones","doi":"10.1145/3441501.3441677","DOIUrl":"https://doi.org/10.1145/3441501.3441677","url":null,"abstract":"We present a real world case study for the evaluation of professional search focusing on German construction law. Reliable identification of relevant previous cases is an important part of many legal disputes, and currently relies on domain expertise acquired over a lengthy professional career. We describe our experiences from the development of a Cranfield type test collection for a German construction law dataset to enable research into the development of search technologies for new tools which are less dependent on expert knowledge. We describe examination of the search needs of lawyers, the development of a set of search queries created by lawyers, and our experiences in collecting expert relevance data for the completion of a test collection for legal search. Important findings of this latter process are the need for individuals with expert legal training to assess relevance, and the identification of context dependence in determining relevance. While the cost of the development of this test collection was found to be very high, we demonstrate its value in terms of identifying the effectiveness of legal search methods and in identifying research directions for legal case search.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131233519","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Finding legal concepts pertaining to court case judgement documents, is an important task in the field of legal data mining. These concepts are also popularly termed as catchwords/keywords. Existing methods for the task lack the ability to extract legal concepts that may not explicitly be mentioned in the document, but present abstractly. This is because the methods do not incorporate legal domain specific information. In this paper, we propose the use of Statutes to solve this task. Evaluation on a set of 1200 Indian Supreme Court Case Documents suggest the effectiveness of our approach, and opens the possibilities of exploring more in this direction.
{"title":"Unsupervised Legal Concept Extraction from Indian Case Documents using Statutes","authors":"Riya Sanjay Podder, Paheli Bhattacharya","doi":"10.1145/3441501.3441508","DOIUrl":"https://doi.org/10.1145/3441501.3441508","url":null,"abstract":"Finding legal concepts pertaining to court case judgement documents, is an important task in the field of legal data mining. These concepts are also popularly termed as catchwords/keywords. Existing methods for the task lack the ability to extract legal concepts that may not explicitly be mentioned in the document, but present abstractly. This is because the methods do not incorporate legal domain specific information. In this paper, we propose the use of Statutes to solve this task. Evaluation on a set of 1200 Indian Supreme Court Case Documents suggest the effectiveness of our approach, and opens the possibilities of exploring more in this direction.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129286419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. Mahata, Subhabrata Dutta, Dipankar Das, Sivaji Bandyopadhyay
Translation systems require a huge amount of parallel data to produce quality translations, but acquiring one for low-resource languages is difficult. To counter this, recent research has been shown to combine languages and use them to augment the low resource data, through transfer learning. While the gain in performance is apparent using transfer learning, we try to investigate the correlation between the performance gain and position of the concerned languages within a language family. We further probe and try to coordinate the performance gain with the degree of vocabulary sharing between the concerned languages.
{"title":"Performance Gain in Low Resource MT with Transfer Learning: An Analysis concerning Language Families","authors":"S. Mahata, Subhabrata Dutta, Dipankar Das, Sivaji Bandyopadhyay","doi":"10.1145/3441501.3441507","DOIUrl":"https://doi.org/10.1145/3441501.3441507","url":null,"abstract":"Translation systems require a huge amount of parallel data to produce quality translations, but acquiring one for low-resource languages is difficult. To counter this, recent research has been shown to combine languages and use them to augment the low resource data, through transfer learning. While the gain in performance is apparent using transfer learning, we try to investigate the correlation between the performance gain and position of the concerned languages within a language family. We further probe and try to coordinate the performance gain with the degree of vocabulary sharing between the concerned languages.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"70 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128934381","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Extraction of causal relations from text is an important problem in Natural Language Processing (NLP). The extracted relations play important roles in several downstream analytical and predictive tasks like identification of actionable items, question-answering and isolation of predictor variables for a predictive system. Curating causal relations from text documents can also help in automatically building causal networks which are also useful for reasoning tasks. The proposed CEREX track aims to find a suitable model for automatic detection of causal sentences and extraction of the exact cause, effect and the causal connectives from textual mentions.
{"title":"CEREX@FIRE-2020: Overview of the Shared Task on Cause-effect Relation Extraction","authors":"Manjira Sinha, Tirthankar Dasgupta, Lipika Dey","doi":"10.1145/3441501.3441514","DOIUrl":"https://doi.org/10.1145/3441501.3441514","url":null,"abstract":"Extraction of causal relations from text is an important problem in Natural Language Processing (NLP). The extracted relations play important roles in several downstream analytical and predictive tasks like identification of actionable items, question-answering and isolation of predictor variables for a predictive system. Curating causal relations from text documents can also help in automatically building causal networks which are also useful for reasoning tasks. The proposed CEREX track aims to find a suitable model for automatic detection of causal sentences and extraction of the exact cause, effect and the causal connectives from textual mentions.","PeriodicalId":415985,"journal":{"name":"Proceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128434760","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：