Graphical Models最新文献

Today’s Extended Reality (XR) applications that call for specific Diminished Reality (DR) strategies to hide specific classes of objects are increasingly using 360° cameras, which can capture entire areas in a single picture. In this work, we present an interactive-based image processing, editing and rendering system named SPIDER, that takes a spherical 360° indoor scene as input. The system is composed of a novel integrated deep learning architecture for extracting geometric and semantic information of full and empty rooms, based on gated and dilated convolutions, followed by a super-resolution module for improving the resolution of the color and depth signals. The obtained high resolution representations allow users to perform interactive exploration and basic editing operations on the reconstructed indoor scene, namely: (i) rendering of the scene in various modalities (point cloud, polygonal, wireframe) (ii) refurnishing (transferring portions of rooms) (iii) deferred shading through the usage of precomputed normal maps. These kinds of scene editing and manipulations can be used for assessing the inference from deep learning models and enable several Mixed Reality applications in areas such as furniture retails, interior designs, and real estates. Moreover, it can also be useful in data augmentation, arts, designs, and paintings. We report on the performance improvement of the various processing components on public domain spherical image indoor datasets.

We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual features between the hand and the object are entangled, and the reconstruction of the interaction area deformation is difficult. To tackle this challenge, we propose a method to annotate our captured data. Our key idea is to collaborate with estimated hand features to guide the object global pose estimation, and then optimize the deformation process of the object by analyzing the relationship between the hand and the object. Through comprehensive evaluation, the proposed method can reconstruct interactive motions of hands and deformable objects with high quality. HMDO currently consists of 21600 frames over 12 sequences. In the future, this dataset could boost the research of learning-based reconstruction of deformable interaction scenes.

Existing approaches (in particular graph theoretic) for generating floorplans focus on constructing floorplans for given adjacencies without considering boundary layout or room shapes. With recent developments in designs, it is demanding to consider multiple constraints while generating floorplan layouts. In this paper, we study graph theoretic properties which guarantee the presence of different shaped rooms within the floorplans. Further, we present a graph-algorithms based application, developed in Python, for generating floorplans with given input room shapes. The proposed application is useful in creating floorplans for a given graph with desired room shapes mainly, L, T, F, C, staircase, and plus-shape. Here, the floorplan boundary is always rectangular. In future,we aim to extend this work to generate any (rectilinear) room shape and floor plan boundary for a given graph.

Most existing methods for RGB hand pose estimation use root-relative 3D coordinates for supervision. However, such supervision neglects the distance between the camera and the object (i.e., the hand). The camera distance is especially important under a perspective camera, which controls the depth-dependent scaling of the perspective projection. As a result, the same hand pose, with different camera distances can be projected into different 2D shapes by the same perspective camera. Neglecting such important information results in ambiguities in recovering 3D poses from 2D images. In this article, we propose a camera projection learning module (CPLM) that uses the scale factor contained in the camera distance to associate 3D hand pose with 2D UV coordinates, which facilities to further optimize the accuracy of the estimated hand joints. Specifically, following the previous work, we use a two-stage RGB-to-2D and 2D-to-3D method to estimate 3D hand pose and embed a graph convolutional network in the second stage to leverage the information contained in the complex non-Euclidean structure of 2D hand joints. Experimental results demonstrate that our proposed method surpasses state-of-the-art methods on the benchmark dataset RHD and obtains competitive results on the STB and D+O datasets.

Completing an unordered partial point cloud is a challenging task. Existing approaches that rely on decoding a latent feature to recover the complete shape, often lead to the completed point cloud being over-smoothing, losing details, and noisy. Instead of decoding a whole shape, we propose to decode and refine a low-resolution (low-res) point cloud first, and then perform a patch-wise noise-aware upsampling rather than interpolating the whole sparse point cloud at once, which tends to lose details. Regarding the possibility of lacking details of the initially decoded low-res point cloud, we propose an iterative refinement to recover the geometric details and a symmetrization process to preserve the trustworthy information from the input partial point cloud. After obtaining a sparse and complete point cloud, we propose a patch-wise upsampling strategy. Patch-based upsampling allows to recover fine details better rather than decoding a whole shape. The patch extraction approach is to generate training patch pairs between the sparse and ground-truth point clouds with an outlier removal step to suppress the noisy points from the sparse point cloud. Together with the low-res recovery, our whole pipeline can achieve high-fidelity point cloud completion. Comprehensive evaluations are provided to demonstrate the effectiveness of the proposed method and its components.

Procedural techniques have been successfully utilized for generating various kinds of 3D models. In this study, we propose a procedural method to build 3D towns that can be manipulated by a set of high-level semantic principles namely security, privacy, sustainability, social-life, economy, and beauty. Based on the user defined weights of these principles, our method generates a 3D settlement to accommodate a desired population over a given terrain. Our approach firstly determines where to establish the settlement over the large terrain which is followed by iteratively constructing the town. In both steps, the principles guide the decisions and our method generates natural looking small-scale 3D residential regions similar to the cities of pre-industrial era. We demonstrate the effectiveness of the proposed approach to build semantically plausible town models by presenting sample results over real world based terrains.

Integrating structured-light technique with deep learning for single-shot 3D imaging has recently gained enormous attention due to its unprecedented robustness. This paper presents an innovative technique of supervised learning-based 3D imaging from a single grayscale structured-light image. The proposed approach uses a single-input, double-output convolutional neural network to transform a regular fringe-pattern image into two intermediate quantities which facilitate the subsequent 3D image reconstruction with high accuracy. A few experiments have been conducted to demonstrate the validity and robustness of the proposed technique.

Capturing outdoor visual-inertial datasets is a challenging yet vital aspect of developing robust visual-inertial odometry (VIO) algorithms. A significant hurdle is that high-accuracy-ground-truth systems (e.g., motion capture) are not practical for outdoor use. One solution is to use a “semi-synthetic” approach that combines rendered images with real IMU data. This approach can produce sequences containing challenging imagery and accurate ground truth but with less simulated data than a fully synthetic sequence. Existing methods (used by popular tools/datasets) record IMU measurements from a visual-inertial system while measuring its trajectory using motion capture, then rendering images along that trajectory. This work identifies a major flaw in that approach, specifically that using motion capture alone to estimate the pose of the robot/system results in the generation of inconsistent visual-inertial data that is not suitable for evaluating VIO algorithms. However, we show that it is possible to generate high-quality semi-synthetic data for VIO algorithm evaluation. We do so using an open-source full-batch optimisation tool to incorporate both mocap and IMU measurements when estimating the IMU’s trajectory. We demonstrate that this improved trajectory results in better consistency between the IMU data and rendered images and that the resulting data improves VIO trajectory error by 79% compared to existing methods. Furthermore, we examine the effect of visual-inertial data inconsistency (as a result of trajectory noise) on VIO performance to provide a foundation for future work targeting real-time applications.

We propose a new volume reconstruction technique based on the six-direction cubic box-spline $M_6$. $M_6$ is $C^1$ continuous and possesses an approximation order of three, the same as that of the tri-quadratic B-spline but with much lower degree. In fact, $M_6$ has the lowest degree among the symmetric box-splines on $Z^3$ with at least $C^1$ continuity. We analyze the polynomial structure induced by the shifts of $M_6$ and propose an efficient analytic evaluation algorithm for splines and their derivatives (gradient and Hessian) based on the high symmetry of $M_6$. To verify the evaluation algorithm, we implement a real-time GPU (graphics processing unit) isosurface raycaster which exhibits interactive performance (54.5 fps (frames per second) with $241^3$ dataset on $512^2$ framebuffer) on a modern graphics hardware. Moreover, we analyze $M_6$ as a reconstruction filter and state that it is comparable to the tri-cubic B-spline, which possesses a higher approximation order.

The normal vector is a basic attribute of point clouds. Traditional estimation methods are susceptible to noise and outliers. Recently, it reported that estimation robustness can be greatly improved by introducing Deep Neural Network (DNN), but how to accurately obtain the normal vector of sharp features still needs to be further studied. This paper proposes SharpNet, a DNN framework specializing in sharp features of CAD-like models, to transform problems into feature classification by the discretization of normal vector space. In order to eliminate the discretization error, a normal vector refining method is presented, which uses the difference between the initial normal vectors to distinguish neighborhood points of different local surface patches. Finally, the normal vector can be estimated accurately from the refined neighborhood points. Experiments show that our algorithm can estimate the normal vector of sharp features of CAD-like models accurately in challenging situations, and is superior to other DNN-based methods in terms of efficiency.