Advances in Data Analysis and Classification最新文献

The use of hyperspectral imaging to investigate food samples has grown due to the improved performance and lower cost of instrumentation. Food engineers use hyperspectral images to classify the type and quality of a food sample, typically using classification methods. In order to train these methods, every pixel in each training image needs to be labelled. Typically, computationally cheap threshold-based approaches are used to label the pixels, and classification methods are trained based on those labels. However, threshold-based approaches are subjective and cannot be generalized across hyperspectral images taken in different conditions and of different foods. Here a consensus-constrained parsimonious Gaussian mixture model (ccPGMM) is proposed to label pixels in hyperspectral images using a model-based clustering approach. The ccPGMM utilizes information that is available on some pixels and specifies constraints on those pixels belonging to the same or different clusters while clustering the rest of the pixels in the image. A latent variable model is used to represent the high-dimensional data in terms of a small number of underlying latent factors. To ensure computational feasibility, a consensus clustering approach is employed, where the data are divided into multiple randomly selected subsets of variables and constrained clustering is applied to each data subset; the clustering results are then consolidated across all data subsets to provide a consensus clustering solution. The ccPGMM approach is applied to simulated datasets and real hyperspectral images of three types of puffed cereal, corn, rice, and wheat. Improved clustering performance and computational efficiency are demonstrated when compared to other current state-of-the-art approaches.

The class of Spatial Dynamic Panel Data models has been proposed in the socio-econometric literature to analyze spatio-temporal data. In this paper we consider a particular variant of such models, where the set of spatial units is assumed to be partitioned into clusters and the parameters of the model are assumed to be homogeneous within clusters and heterogeneous across clusters. For this model, assuming that the true partition is unknown, we propose a new clustering procedure and a validation test, based on a multiple testing approach, that help to choose the best configuration of model, for a given observed dataset, by estimating the optimal number of clusters and the best partition of units. The validity of the proposed procedures has been shown both theoretically and empirically, on simulated and real data, also compared to alternative methods.

Hierarchical categorical variables often exhibit many levels (high granularity) and many classes within each level (high dimensionality). This may cause overfitting and estimation issues when including such covariates in a predictive model. In current literature, a hierarchical covariate is often incorporated via nested random effects. However, this does not facilitate the assumption of classes having the same effect on the response variable. In this paper, we propose a methodology to obtain a reduced representation of a hierarchical categorical variable. We show how entity embedding can be applied in a hierarchical setting. Subsequently, we propose a top-down clustering algorithm which leverages the information encoded in the embeddings to reduce both the within-level dimensionality as well as the overall granularity of the hierarchical categorical variable. In simulation experiments, we show that our methodology can effectively approximate the true underlying structure of a hierarchical covariate in terms of the effect on a response variable, and find that incorporating the reduced hierarchy improves the balance between model fit and complexity. We apply our methodology to a real dataset and find that the reduced hierarchy is an improvement over the original hierarchical structure and reduced structures proposed in the literature.

Clustering univariate functional data is mostly based on projecting the curves onto an adequate basis and applying some distance or similarity models on the coefficients. The basis functions should be chosen depending on features of the function being estimated. Commonly used are Fourier, polynomial and splines, but these may not be well suited for curves that exhibit inhomogeneous behavior. Wavelets on the contrary are well suited for identifying highly discriminant local time and scale features, and are able to adapt to the data smoothness. In recent years, few methods, relying on wavelet-based similarity measures, have been proposed for clustering curves, observed on equidistant points. In this work, we present a non-equidistant design wavelet based method for non-parametrically estimating and clustering a large number of curves. The method consists of several crucial stages: fitting functional data by non-equispaced design wavelet regression, screening out nearly flat curves, denoising the remaining curves with wavelet thresholding, and finally clustering the denoised curves. Simulation studies compare our proposed method with some other functional clustering methods. The method is applied for clustering some real functional data profiles.

The goal of functional clustering is twofold: first, to categorize curves with similar temporal behaviors into separate clusters, and second, to obtain a representative curve that summarizes the typical temporal behavior within each cluster. An important challenge in current functional clustering techniques is class imbalance, where some clusters contain a significantly greater number of curves than others. While class imbalance is extensively addressed in supervised classification, it remains relatively unexplored in unsupervised contexts. To address this gap, we propose adapting the iterative hierarchical clustering approach, originally designed for multivariate data, to the context of functional data. Thus introducing a novel method called functional iterative hierarchical clustering (funIHC) to effectively handle the clustering of imbalanced functional data. Through comprehensive simulation studies and benchmarking datasets, we demonstrate the effectiveness of the funIHC approach. Utilizing funIHC on gene expression data related to human influenza infection induced by the H3N2 virus, we identify five distinct and biologically meaningful patterns of gene expression. The R and MATLAB code for implementing funIHC is freely accessible at www.fdaatucd.com.

Multiverse Analysis is a heuristic for robust multiple models estimation where data fit many connected specifications of the same abstract model, instead of a singular or a small selection of specifications. Differently from the canonical application of multimodels, in Multiverse Analysis the probabilities of the specifications to be included in the analysis are never assumed independent of each other. Grounded in this consideration, this study provides a compact statistical characterisation of the process of elicitation of the specifications in Multiverse Analysis and conceptually adjacent methods, connecting previous insights from meta-analytical Statistics, model averaging, Network Theory, Information Theory, and Causal Inference. The calibration of the multiversal estimates is treated with references to the adoption of Bayesian Model Averaging vs. alternatives. In the applications, it is checked the theory that Bayesian Model Averaging reduces both error and uncertainty for well-specified multiversal models but amplifies errors when a collider variable is included in the multiversal model. In well-specified models, alternatives do not perform better than Uniform weighting of the estimates, so the adoption of a gold standard remains ambiguous. Normative implications for misinterpretation of Multiverse Analysis and future directions of research are discussed.

Over the last decades, co-clustering models have spawned a number of algorithms showing the advantages that co-clustering can have over clustering. This is especially true for sparse high-dimensional data such as document-word matrices, which are our focus here. This proposal uses Latent Block Models (LBMs), rigorous statistical models that offer a variety of benefits in terms of flexibility, parsimony, and effectiveness. LBMs have been proposed in relation to different data types. This paper aims to embed existing and new models in a unified framework, focusing on exponential family LBM (ELBM) and the classification maximum likelihood approach. We then extend these models to include sparse versions, known as SELBM, taking into account the sparsity of datasets. The matrix formulations that we propose lead to simplified algorithms capable of addressing various types of data effectively.

Statistical analysis of censored data has received considerable interest across various fields, including biomedical, clinical and econometrical studies. The cause of censored measurement is usually induced by the limitation of measuring instruments and/or experimental design. In practice, regression modeling for censored data might encounter departure from normality of errors due primarily to the latent sources of heterogeneity, and/or the presence of atypical observations and outliers. A Bayesian analysis for the mixture of linear experts model is studied wherein the errors follow the scale mixture of normal distribution, and the responses suffer from either a left or right censoring schemes. We propose a weakly informative prior structure for the parameters and show that the corresponding posterior distributions are proper. Leveraging the Ultimate Pólya-Gamma data-augmentation method, we efficiently sample gating parameters and consequentially allocate cluster memberships. Compared to the traditional maximum likelihood method, our Bayesian approach is shown to mitigate the impact of censoring on deteriorating estimation and classification abilities. The effectiveness of our proposal is illustrated by undertaking some synthetic studies and a real data example. (texttt{R}) scripts for the implementation of our Bayesian methods are available at the (texttt{GitHub}) repository.