Acquiring manually labeled training data for a specific application is expensive and while such data is often fully available for casual camera imagery, it is not a good fit for novel cameras. To overcome this, we present a repurposing approach that relies on spherical image warping to retarget an existing dataset of landmark labeled casual photography of people's faces with arbitrary poses from regular camera lenses to target cameras with significantly different intrinsics, such as those often attached to the head mounted displays (HMDs) with wide-angle lenses necessary to observe mouth and other features at close proximity and infrared only sensing for eye observations. Our method can predict landmarks of the HMD wearer in facial sub-regions in a divide-and-conquer fashion with particular focus on mouth and eyes. We demonstrate animated avatars in realtime using the face landmarks as input without user-specific nor application-specific dataset.
{"title":"Repurposing Labeled Photographs for Facial Tracking with Alternative Camera Intrinsics","authors":"Caio Brito, Kenny Mitchell","doi":"10.1109/VR.2019.8798303","DOIUrl":"https://doi.org/10.1109/VR.2019.8798303","url":null,"abstract":"Acquiring manually labeled training data for a specific application is expensive and while such data is often fully available for casual camera imagery, it is not a good fit for novel cameras. To overcome this, we present a repurposing approach that relies on spherical image warping to retarget an existing dataset of landmark labeled casual photography of people's faces with arbitrary poses from regular camera lenses to target cameras with significantly different intrinsics, such as those often attached to the head mounted displays (HMDs) with wide-angle lenses necessary to observe mouth and other features at close proximity and infrared only sensing for eye observations. Our method can predict landmarks of the HMD wearer in facial sub-regions in a divide-and-conquer fashion with particular focus on mouth and eyes. We demonstrate animated avatars in realtime using the face landmarks as input without user-specific nor application-specific dataset.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132818080","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Existing virtual reality (VR) based wheelchair simulators have difficulty providing both visual and motion feedback at low cost. To address this issue, we propose a VR-based wheelchair simulator using a combination of motions attainable by an electric-powered wheelchair and vection-inducing movies displayed on a head-mounted display. This approach enables the user to have a richer simulation experience, because the scenes of the movie change as if the wheelchair performs motions that are not actually performable. We developed a proof of concept using only consumer products and conducted evaluation tasks, confirming that our approach can provide a richer experience for barrier simulations.
{"title":"Evaluation on a Wheelchair Simulator Using Limited-Motion Patterns and Vection-Inducing Movies","authors":"Akihiro Miyata, Hironobu Uno, Kenro Go","doi":"10.1109/VR.2019.8797726","DOIUrl":"https://doi.org/10.1109/VR.2019.8797726","url":null,"abstract":"Existing virtual reality (VR) based wheelchair simulators have difficulty providing both visual and motion feedback at low cost. To address this issue, we propose a VR-based wheelchair simulator using a combination of motions attainable by an electric-powered wheelchair and vection-inducing movies displayed on a head-mounted display. This approach enables the user to have a richer simulation experience, because the scenes of the movie change as if the wheelchair performs motions that are not actually performable. We developed a proof of concept using only consumer products and conducted evaluation tasks, confirming that our approach can provide a richer experience for barrier simulations.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131119630","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Manuela Chessa, Guido Maiello, Lina K. Klein, Vivian C. Paulun, F. Solari
Grasping is one of the fundamental actions we perform to interact with objects in real environments, and in the real world we rarely experience difficulty picking up objects. Grasping plays a fundamental role for interactive virtual reality (VR) systems that are increasingly employed not only for recreational purposes, but also for training in industrial contexts, in medical tasks, and for rehabilitation protocols. To ensure the effectiveness of such VR applications, we must understand whether the same grasping behaviors and strategies employed in the real world are adopted when interacting with objects in VR. To this aim, we replicated in VR an experimental paradigm employed to investigate grasping behavior in the real world. We tracked participants' forefinger and thumb as they picked up, in a VR environment, unfamiliar objects presented at different orientations, and exhibiting the same physics behavior of their real counterparts. We compared grasping behavior within and across participants, in VR and in the corresponding real world situation. Our findings highlight the similarities and differences in grasping behavior in real and virtual environments.
{"title":"Grasping objects in immersive Virtual Reality","authors":"Manuela Chessa, Guido Maiello, Lina K. Klein, Vivian C. Paulun, F. Solari","doi":"10.1109/VR.2019.8798155","DOIUrl":"https://doi.org/10.1109/VR.2019.8798155","url":null,"abstract":"Grasping is one of the fundamental actions we perform to interact with objects in real environments, and in the real world we rarely experience difficulty picking up objects. Grasping plays a fundamental role for interactive virtual reality (VR) systems that are increasingly employed not only for recreational purposes, but also for training in industrial contexts, in medical tasks, and for rehabilitation protocols. To ensure the effectiveness of such VR applications, we must understand whether the same grasping behaviors and strategies employed in the real world are adopted when interacting with objects in VR. To this aim, we replicated in VR an experimental paradigm employed to investigate grasping behavior in the real world. We tracked participants' forefinger and thumb as they picked up, in a VR environment, unfamiliar objects presented at different orientations, and exhibiting the same physics behavior of their real counterparts. We compared grasping behavior within and across participants, in VR and in the corresponding real world situation. Our findings highlight the similarities and differences in grasping behavior in real and virtual environments.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"205 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133134927","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Veronez, L. G. D. Silveira, F. Bordin, Leonardo Campos Inocencio, Graciela Racolte, L. S. Kupssinskü, Pedro Rossa, L. Scalco
One of the main difficulties in the inspection of Bridges/Viaducts by observation is inaccessibility or lack of access throughout the structure. Mapping using remote sensors on Unmanned Aerial Vehicles (UAVs) or by means of laser scanning can be an interesting alternative to the engineer as it can enable more detailed analysis and diagnostics. Such mapping techniques also allow the generation of realistic 3D models that can be integrated in Virtual Reality (VR) environments. In this sense, we present the ImSpector, a system that uses realistic 3D models generated by remote sensors embedded in UAVs that implements a virtual and immersive environment for inspections. As a result, the system provides the engineer a tool to carry out field tests directly at the office, ensuring agility, accuracy and safety in bridge and viaduct inspections.
{"title":"Imspector: Immersive System of Inspection of Bridges/Viaducts","authors":"M. Veronez, L. G. D. Silveira, F. Bordin, Leonardo Campos Inocencio, Graciela Racolte, L. S. Kupssinskü, Pedro Rossa, L. Scalco","doi":"10.1109/VR.2019.8798295","DOIUrl":"https://doi.org/10.1109/VR.2019.8798295","url":null,"abstract":"One of the main difficulties in the inspection of Bridges/Viaducts by observation is inaccessibility or lack of access throughout the structure. Mapping using remote sensors on Unmanned Aerial Vehicles (UAVs) or by means of laser scanning can be an interesting alternative to the engineer as it can enable more detailed analysis and diagnostics. Such mapping techniques also allow the generation of realistic 3D models that can be integrated in Virtual Reality (VR) environments. In this sense, we present the ImSpector, a system that uses realistic 3D models generated by remote sensors embedded in UAVs that implements a virtual and immersive environment for inspections. As a result, the system provides the engineer a tool to carry out field tests directly at the office, ensuring agility, accuracy and safety in bridge and viaduct inspections.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"275 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123497461","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Wan-Lun Tsai, Liwei Su, Tsai-Yen Ko, Cheng-Ta Yang, Min-Chun Hu
Decision-making is an essential part in basketball offenses. In this paper, we proposed a basketball offensive decision-making VR training system. During the training, the trainee can intuitively interact with the system by wearing a motion capture suit and be trained in different virtual defensive scenarios designed by professional coaches. The system will recognize the offensive action performed by the user and provide correct suggestions when he/she makes a bad offensive decision. We compared the effectiveness of the training protocols by using conventional tactics board and the proposed VR system. Furthermore, we investigated the influence of using prerecorded 360-degree panorama video and computer simulated virtual content to create immersive training environment.
{"title":"Improve the Decision-making Skill of Basketball Players by an Action-aware VR Training System","authors":"Wan-Lun Tsai, Liwei Su, Tsai-Yen Ko, Cheng-Ta Yang, Min-Chun Hu","doi":"10.1109/VR.2019.8798309","DOIUrl":"https://doi.org/10.1109/VR.2019.8798309","url":null,"abstract":"Decision-making is an essential part in basketball offenses. In this paper, we proposed a basketball offensive decision-making VR training system. During the training, the trainee can intuitively interact with the system by wearing a motion capture suit and be trained in different virtual defensive scenarios designed by professional coaches. The system will recognize the offensive action performed by the user and provide correct suggestions when he/she makes a bad offensive decision. We compared the effectiveness of the training protocols by using conventional tactics board and the proposed VR system. Furthermore, we investigated the influence of using prerecorded 360-degree panorama video and computer simulated virtual content to create immersive training environment.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123702165","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We propose an approach to facilitate adjustable grip for object interaction in virtual reality. It enables the user to handle objects with loose and firm grip using conventional controllers. Pivotal design properties were identified and evaluated in a qualitative pilot study. Two revised interaction designs with variable grip were compared to the status quo of invariable grip in a quantitative study. The users performed placing actions with all interaction modes. Performance, clutching, task load, and usability were measured. While the handling time increased slightly using variable grip, the usability score was significantly higher. No substantial differences were measured in positioning accuracy. The results lead to the conclusion that variable grip can be useful and improve realism depending on tasks, goals, and user preference.
{"title":"Get a Grip! Introducing Variable Grip for Controller-Based VR Systems","authors":"Michael Bonfert, R. Porzel, R. Malaka","doi":"10.1109/VR.2019.8797824","DOIUrl":"https://doi.org/10.1109/VR.2019.8797824","url":null,"abstract":"We propose an approach to facilitate adjustable grip for object interaction in virtual reality. It enables the user to handle objects with loose and firm grip using conventional controllers. Pivotal design properties were identified and evaluated in a qualitative pilot study. Two revised interaction designs with variable grip were compared to the status quo of invariable grip in a quantitative study. The users performed placing actions with all interaction modes. Performance, clutching, task load, and usability were measured. While the handling time increased slightly using variable grip, the usability score was significantly higher. No substantial differences were measured in positioning accuracy. The results lead to the conclusion that variable grip can be useful and improve realism depending on tasks, goals, and user preference.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"70 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125259349","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Recent advances of the affordable virtual reality headsets make virtual reality training an economical choice when compared to traditional training. However, these virtual reality devices present a range of different levels of virtual reality fidelity and interactions. Few works have evaluated their validity against the traditional training formats. This paper presents a study that compares the learning efficiency of a bimanual gearbox assembly task among traditional training, virtual reality training with direct 3D inputs (HTC VIVE), and virtual reality training without 3D inputs (Google Cardboard). A pilot study was conducted and the result shows that HTC VIVE brings the best learning outcomes.
{"title":"Training Transfer of Bimanual Assembly Tasks in Cost-Differentiated Virtual Reality Systems","authors":"S. Shen, Hsiang-Ting Chen, T. Leong","doi":"10.1109/VR.2019.8797917","DOIUrl":"https://doi.org/10.1109/VR.2019.8797917","url":null,"abstract":"Recent advances of the affordable virtual reality headsets make virtual reality training an economical choice when compared to traditional training. However, these virtual reality devices present a range of different levels of virtual reality fidelity and interactions. Few works have evaluated their validity against the traditional training formats. This paper presents a study that compares the learning efficiency of a bimanual gearbox assembly task among traditional training, virtual reality training with direct 3D inputs (HTC VIVE), and virtual reality training without 3D inputs (Google Cardboard). A pilot study was conducted and the result shows that HTC VIVE brings the best learning outcomes.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125156824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper discusses a virtual reality (VR) therapeutic video game for treatment of the neurological eye disorder, Amblyopia. Amblyopia is often referred to as lazy eye, and it entails weaker vision in one eye due to a poor connection between the eye and the brain. Until recently it was thought to be untreatable in adults, but new research has proven that with consistent therapy even adults can improve their Amblyopia, especially through perceptual learning and video games. Even so, therapy compliance remains low due to the fact that conventional therapies are perceived as either invasive, dull and/or boring. Our game aims to make Amblyopia therapy more immersive, enjoyable and playful. The game was perceived by our users to be a fun and accessible alternative, as it involves adhering a Bangerter foil (an opaque sticker) on a VR headset to blur vision in an Amblyopic person's dominant eye while having them playa VR video game. To perform well in the video game, their brain must adapt to rely on seeing with their weaker eye, thereby reforging that neurological connection. While testing our game, we also studied users behavior to investigate what visual and kinetic components were more effective therapeutically. Our findings generally show positive results, showing that visual acuity in adults increases with 45 minutes of therapy. Amblyopia has many negative symptoms including poor depth perception (nec-essary for daily activities such as driving), so this therapy could be life changing for adults with Amblyopia.
{"title":"Virtual Reality Video Game Paired with Physical Monocular Blurring as Accessible Therapy for Amblyopia","authors":"O. Hurd, S. Kurniawan, M. Teodorescu","doi":"10.1109/VR.2019.8797997","DOIUrl":"https://doi.org/10.1109/VR.2019.8797997","url":null,"abstract":"This paper discusses a virtual reality (VR) therapeutic video game for treatment of the neurological eye disorder, Amblyopia. Amblyopia is often referred to as lazy eye, and it entails weaker vision in one eye due to a poor connection between the eye and the brain. Until recently it was thought to be untreatable in adults, but new research has proven that with consistent therapy even adults can improve their Amblyopia, especially through perceptual learning and video games. Even so, therapy compliance remains low due to the fact that conventional therapies are perceived as either invasive, dull and/or boring. Our game aims to make Amblyopia therapy more immersive, enjoyable and playful. The game was perceived by our users to be a fun and accessible alternative, as it involves adhering a Bangerter foil (an opaque sticker) on a VR headset to blur vision in an Amblyopic person's dominant eye while having them playa VR video game. To perform well in the video game, their brain must adapt to rely on seeing with their weaker eye, thereby reforging that neurological connection. While testing our game, we also studied users behavior to investigate what visual and kinetic components were more effective therapeutically. Our findings generally show positive results, showing that visual acuity in adults increases with 45 minutes of therapy. Amblyopia has many negative symptoms including poor depth perception (nec-essary for daily activities such as driving), so this therapy could be life changing for adults with Amblyopia.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116101938","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Varifocal displays are a practical method to solve vergence–accommodation conflict in near-eye displays for both virtual and augmented reality, but they are reliant on knowing the user's focal state. One approach for detecting the focal state is to use the link between vergence and accommodation and employ binocular gaze tracking to determine the depth of the fixation point; consequently, the focal depth is also known. In order to ensure the virtual image is in focus, the display must be set to a depth which causes no negative perceptual or physiological effects to the viewer, which indicates error bounds for the calculation of fixation point. I analyze the required gaze tracker accuracy to ensure the display focus is set within the viewer's depth of field, zone of comfort, and zone of clear single binocular vision. My findings indicate that for the median adult using an augmented reality varifocal display, gaze tracking accuracy must be better than 0.541°. In addition, I discuss eye tracking approaches presented in the literature to determine their ability to meet the specified requirements.
{"title":"Required Accuracy of Gaze Tracking for Varifocal Displays","authors":"David Dunn","doi":"10.1109/VR.2019.8798273","DOIUrl":"https://doi.org/10.1109/VR.2019.8798273","url":null,"abstract":"Varifocal displays are a practical method to solve vergence–accommodation conflict in near-eye displays for both virtual and augmented reality, but they are reliant on knowing the user's focal state. One approach for detecting the focal state is to use the link between vergence and accommodation and employ binocular gaze tracking to determine the depth of the fixation point; consequently, the focal depth is also known. In order to ensure the virtual image is in focus, the display must be set to a depth which causes no negative perceptual or physiological effects to the viewer, which indicates error bounds for the calculation of fixation point. I analyze the required gaze tracker accuracy to ensure the display focus is set within the viewer's depth of field, zone of comfort, and zone of clear single binocular vision. My findings indicate that for the median adult using an augmented reality varifocal display, gaze tracking accuracy must be better than 0.541°. In addition, I discuss eye tracking approaches presented in the literature to determine their ability to meet the specified requirements.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127208962","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
E. Wisotzky, Jean-Claude Rosenthal, P. Eisert, A. Hilsmann, Falko Schmid, M. Bauer, Armin Schneider, F. Uecker
We present an interactive and multimodal-based augmented reality system for computer-assisted surgery in the context of ear, nose and throat (ENT) treatment. The proposed processing pipeline uses fully digital stereoscopic imaging devices, which support multispectral and white light imaging to generate high resolution image data, and consists of five modules. Input/output data handling, a hybrid multimodal image analysis and a bi-directional interactive augmented reality (AR) and mixed reality (MR) interface for local and remote surgical assistance are of high relevance for the complete framework. The hybrid multimodal 3D scene analysis module uses different wavelengths to classify tissue structures and combines this spectral data with metric 3D information. Additionally, we propose a zoom-independent intraoperative tool for virtual ossicular prosthesis insertion (e.g. stapedectomy) guaranteeing very high metric accuracy in sub-millimeter range (1/10 mm). A bi-directional interactive AR/MR communication module guarantees low latency, while consisting surgical information and avoiding informational overload. Display agnostic AR/MR visualization can show our analyzed data synchronized inside the digital binocular, the 3D display or any connected head-mounted-display (HMD). In addition, the analyzed data can be enriched with annotations by involving external clinical experts using AR/MR and furthermore an accurate registration of preoperative data. The benefits of such a collaborative surgical system are manifold and will lead to a highly improved patient outcome through an easier tissue classification and reduced surgery risk.
{"title":"Interactive and Multimodal-based Augmented Reality for Remote Assistance using a Digital Surgical Microscope","authors":"E. Wisotzky, Jean-Claude Rosenthal, P. Eisert, A. Hilsmann, Falko Schmid, M. Bauer, Armin Schneider, F. Uecker","doi":"10.1109/VR.2019.8797682","DOIUrl":"https://doi.org/10.1109/VR.2019.8797682","url":null,"abstract":"We present an interactive and multimodal-based augmented reality system for computer-assisted surgery in the context of ear, nose and throat (ENT) treatment. The proposed processing pipeline uses fully digital stereoscopic imaging devices, which support multispectral and white light imaging to generate high resolution image data, and consists of five modules. Input/output data handling, a hybrid multimodal image analysis and a bi-directional interactive augmented reality (AR) and mixed reality (MR) interface for local and remote surgical assistance are of high relevance for the complete framework. The hybrid multimodal 3D scene analysis module uses different wavelengths to classify tissue structures and combines this spectral data with metric 3D information. Additionally, we propose a zoom-independent intraoperative tool for virtual ossicular prosthesis insertion (e.g. stapedectomy) guaranteeing very high metric accuracy in sub-millimeter range (1/10 mm). A bi-directional interactive AR/MR communication module guarantees low latency, while consisting surgical information and avoiding informational overload. Display agnostic AR/MR visualization can show our analyzed data synchronized inside the digital binocular, the 3D display or any connected head-mounted-display (HMD). In addition, the analyzed data can be enriched with annotations by involving external clinical experts using AR/MR and furthermore an accurate registration of preoperative data. The benefits of such a collaborative surgical system are manifold and will lead to a highly improved patient outcome through an easier tissue classification and reduced surgery risk.","PeriodicalId":315935,"journal":{"name":"2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115287830","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：