Annals of Data Science最新文献

Partial label learning (PLL) is a particular problem setting within weakly supervised learning. In PLL, each sample corresponds to a candidate label set in which only one label is true. However, in some practical application scenarios, the emergence of label noise can make some candidate sets lose their true labels, leading to a decline in model performance. In this work, a robust training strategy for PLL, derived from the joint training with co-regularization (JoCoR), is proposed to address this issue in PLL. Specifically, the proposed approach constructs two separate PLL models and a joint loss. The joint loss consists of not only two PLL losses but also a co-regularization term measuring the disagreement of the two models. By automatically selecting samples with small joint loss and using them to update the two models, our proposed approach is able to filter more and more suspected samples with noise candidate label sets. Gradually, the robustness of the PLL models to label noise strengthens due to the reduced disagreement of the two models. Experiments are conducted on two state-of-the-art PLL models using benchmark datasets under various noise levels. The results show that the proposed method can effectively stabilize the training process and reduce the model's overfitting to noisy candidate label sets.

In this paper, a nonparametric kernel method is introduced to estimate the well-known overlapping coefficient, Matusita (rho (X,Y)), between two random variables (X) and (Y). Due to the complexity of finding the formula expression of this coefficient when using the kernel estimators, we suggest to use the numerical integration method to approximate its integral as a first step. Then the kernel estimators were combined with the new approximation to formulate the proposed estimators. Two numerical integration rules known as trapezoidal and Simpson rules were used to approximate the interesting integral. The proposed technique produces two new estimators for (rho (X,Y)). The resulting estimators are studied and compared with existing estimator developed by Eidous and Al-Talafheh (Commun Stat Simul Comput 51(9):5139–5156, 2022. https://doi.org/10.1080/03610918.2020.1757711) via Monte-Carlo simulation technique. The simulation results demonstrated the usefulness and effectiveness of the new technique for estimating (rho (X,Y)).

In this paper we first define the class of Generalized Inflated Power Series Distributions (GIPSDs) which contain the inflated discrete distributions most often seen in practice as special cases. We describe the hitherto unkown exponential family structure of GIPSDs and use this to derive closed-form, easy to program, conditional and unconditional maximum likelihood estimators for essentially any number of parameters. We also show how the GIPSD exponential family can be extended to model deflated mass points. Our results provide easy access to likelihood-based inference and automated model selection procedures for GIPSDs that only involve one-dimensional numerical root-finding problems that are easily solved with simple routines. We consider four real-data examples which illustrate the utility and scope of our results.

With the widespread use of social networks, detecting the topics discussed on these platforms has become a significant challenge. Current approaches primarily rely on frequent pattern mining or semantic relations, often neglecting the structure of the language. Language structural methods aim to discover the relationships between words and how humans understand them. Therefore, this paper introduces a topic detection framework for social networks based on the concept of imitating the mental ability of word association. This framework employs the Human Word Association method and includes a specially designed extraction algorithm. The performance of this method is evaluated using the FA-CUP dataset, a benchmark in the field of topic detection. The results indicate that the proposed method significantly improves topic detection compared to other methods, as evidenced by Topic-recall and the keyword F1 measure. Additionally, to assess the applicability and generalizability of the proposed method, a dataset of Telegram posts in the Persian language is used. The results demonstrate that this method outperforms other topic detection methods.

Agriculture is the primary source of food, fuel, and raw materials and is vital to any country’s economy. Farmers, the backbone of agriculture, primarily rely on instinct to determine what crops to plant in any given season. They are comfortable following customary farming practices and standards and are oblivious to the fact that crop yield is highly dependent on current environmental and soil conditions. Crop recommendations involve multifaceted factors such as weather, soil quality, crop production, market demand, and prices, making it crucial for farmers to make well-informed decisions. An improper or imprudent crop recommendation can affect them, their families, and the entire agricultural sector. Modern technologies like artificial intelligence, machine learning, and data science have emerged as efficient solutions to combat issues like declining crop production and lower profits. This research proposes a Smart Crop Recommendation framework that leverages machine learning to empower farmers to make informed decisions about optimal crop selection. The framework consists of two phases: crop filtration and yield prediction. Crops are filtered in the first phase using an artificial neural network based on local input parameters. The second phase estimates yield for filtered crops, considering the season, farm area, and location data. The final recommendation provides farmers with crops aimed at maximizing profit. The remarkable 99.10% accuracy of the framework is demonstrated through experimentation using artificial neural networks and the 0.99 (text {R}^{text {2}}) error metric for the random forest. The uniqueness of this framework lies in its distinctive focus on the farm level and its consideration of the challenges and various agricultural features that change over time. The experimental results affirm the effectiveness of the framework, and its lightweight nature enhances its practicality, making it an efficient real-time recommendation solution.

Observations indicate that the distributions of stock returns in financial markets usually do not conform to normal distributions, but rather exhibit characteristics of high peaks, fat tails and biases. In this work, we assume that the effects of events or information on prices obey normal distribution, while financial markets often overreact or underreact to events or information, resulting in non normal distributions of stock returns. Based on the above assumptions, we for the first time propose a reaction function for a financial market reacting to events or information, and a model based on it to describe the distribution of real stock returns. Our analysis of the returns of China Securities Index 300 (CSI 300), the Standard & Poor’s 500 Index (SPX or S &P 500) and the Nikkei 225 Index (N225) at different time scales shows that financial markets often underreact to events or information with minor impacts, overreact to events or information with relatively significant impacts, and react slightly stronger to positive events or information than to negative ones. In addition, differences in financial markets and time scales of returns can also affect the shapes of the reaction functions.

In this article, we propose the quadratic rank transmutation map approach on shifted Lindley distribution to improve the existing distribution further. An additional skewness parameter (lambda ) is incorporated to transmute the distribution. The distribution, hence introduced, is called the Transmuted Shifted Lindley distribution. We provide a comprehensive description of this distribution’s statistical properties and its reliability behavior. The heat maps on the associated parameters are presented. In the estimation section, both maximum likelihood and Bayesian estimation of parameters are discussed. A detailed simulation study is performed. Finally, a real data application illustrates the performance of fitting to the proposed distribution.

Automated detection of plant diseases is crucial as it simplifies the task of monitoring large farms and identifies diseases at their early stages to mitigate further plant degradation. Besides the decline in plant health, reduced production severely impacts the country’s economy. Traditional disease identification methods, relying on human experts, are slow, time-consuming, and impractical for large farms. Our proposed model utilizes a combination of pre-trained Resnet18, Alexnet, GoogLeNet, and VGG16 networks to classify apple tree leaves into categories such as healthy, black rot, apple cedar rust, and apple scab based on images. Various image enhancement techniques were employed to enhance the model’s accuracy. Ultimately, our model achieved an accuracy of 97.25% on the validation dataset, demonstrating excellent performance across various metrics. This suggests its potential for efficient and accurate plant health monitoring in the agricultural sector.

In the era of big data, with the increase in volume and complexity of data, the main challenge is how to use big data while preserving the privacy of users. This study was conducted with the aim of finding a solution to this challenge. In this study, we examined various data anonymization methods, including differential privacy, advanced encryption, and strong access controls. In addition, the operation, advantages, disadvantages, and use of these methods, the challenges of adapting these methods to big data, and possible solutions for them were also examined. Our results show that traditional data anonymization methods lack scalability, leading to privacy breaches and data loss. When faced with large volumes of data, these methods may not be able to fully process the data. Also, these methods may be ineffective against re-identification attacks, linkage attacks, and inference attacks. We introduced emerging methods that are capable of providing improved privacy with minimal data loss. These methods have scalability for big data. Finally, we examined future research works and raised important questions that can help improve existing algorithms or develop new methods, better manage the complexity and scale of unstructured data.

In light of the escalating privacy risks in the big data era, this paper introduces an innovative model for the anonymization of big data streams, leveraging in-memory processing within the Spark framework. The approach is founded on the principle of K-anonymity and propels the field forward by critically evaluating various anonymization methods and algorithms, benchmarking their performance with respect to time and space complexities. A distinctive formula for optimized cluster determination in the K-means algorithm is presented, along with a novel tuple expiration time strategy for the efficient purging of clusters. The integration of these components into Spark’s RDD and MLlib modules results in a significant decrease in execution time and data loss rates, even with increasing data volumes. The paper’s notable contributions are its methodological advancements that offer a robust, scalable solution for data anonymization, safeguarding user privacy without sacrificing data utility or processing efficiency.