This project aims to design and model a 3D digital surgeon in the existing ORamaVR’s MAGES SDK platform to guide trainees in virtual surgery. The digital surgeon will identify the trainee’s gestures by recognizing and analyzing the trainee’s hand/arm gestures and provides personalized assessment, real-time feedback, instructions, and recommendations. The proposed system will adopt a distributed software architecture taking advantage of edge-cloud resources and 5G networking to enable ultra-low latency untethered VR.
The digital surgeon’s optimal behavioural model, based on virtual gesture data sets, will be provided by the Hospital of Geneva (HUG). A deep learning model will be derived from recording skilled real surgeons’ gestures to assist the feedback decision engine of the digital surgeon. A distributed VR system architecture will be implemented utilizing edge computing and 5G networks. Furthermore, MAGES system will embed methods of cutting and tearing of physics-based deformable surfaces that advance the state-of-the-art.
University of Geneva, University Hospital of Geneva, ORama S.A., MIRALab sarl.