Anomaly Detection Between Judicial Text-Based Documents

The problem of searching for anomalies or outliers are extremely important in various fields with problems like fraud detection, crime research, network reliability analysis, medical diagnostics etc.What is an anomaly in the judicial system? A court case is to be considered as an anomaly if the judge’s decision differs significantly from existing decisions in similar cases.In most cases, the existing outlier’s search methods use high-dimensional domains in which data can contain hundreds of dimensions. Such an approach requires lots of resources and clearly is not efficient.Objectives: In this article, the authors:•present two methods (or two models) for searching for anomalies in judicial practice;•give a comparative analysis of the results of the effectiveness of both methods.Methodology: The First method for searching for anomalies is a mix of two models: classification and similarity algorithms. Here algorithms like Logistic regression, Extreme Gradient Boosting (XGBoost), Tensorflow for classification and Latent Dirichlet Allocation (LDA), Latent semantic indexing (LSI) to find similar documents. The Second method shows the usage of the Bidirectional Encoder Representations from Transformers (BERT) embedding model and the Annoy indexing model.Findings: The second method shows better and fast results for searching outliers.Data source: Authors used the set of acts provided by the Supreme Court of the Republic of Kazakhstan. The dataset contains 1 million text documents and metadata.