Literature Review on Current Use of Virtual Surgical Intelligence HoloMedicine and Potential Implementation in Trauma and Orthopaedics

Trauma and orthopaedics (T&O) is especially influenced by impacts of technological advancements as operations often require precise preoperative planning and advanced intra-operative skills, which are significant surgeon-related factors in patient outcomes.¹ Amongst different types of current and potential digital application in T&O,² robotics have been recently used as a supplementary surgical tool or an alternative option for conventional surgeries to increase the accuracy and to support the preoperative planning.³ For instance, robotic assisted-total knee arthroplasty (RA-TKA) demonstrated improved accuracy in navigation and minimal soft tissue injuries,⁴ such as collateral ligaments, posterior cruciate ligaments (PCL) or Extensor mechanisms, decreased tibial slope outliers,⁵ higher accuracy in femoral component rotational alignment and decreased loss of blood over traditional TKA.⁶

Despite the benefits of robotic surgeries, the delivery of information is via 2-dimensional (2D) formats such as computer screens. This way of displaying can limit detailed observations in patient movements or evaluation of medical imaging in different angles.⁷

“Holo-medicine”, a combined terminology of “hologram” and “medicine” primarily relies on extended reality (XR) technology, which encompasses virtual reality (VR), augmented reality (AR) and mixed reality (MR).⁸ It is an innovative form of a practice and the delivery of medicine through XR technology transmitting the data to a display device such as head-mounted display (HMD) that projects holographic imageries. VR is a computer-generated depiction of either real or artificial world, whereas AR is a real-world view with computer generated graphics. MR visualises a real-world view combined with an interactive computer-generated graphics.⁹

Virtual surgical intelligence (VSI) HoloMedicine (ApoQlar, Hamburg, Germany) is a MR software running on HMD, HoloLens2 (Microsoft, Redmond, United States). It can visualise medical imaging or documentations in 3-dimensional (3D) holographic imageries surrounding the user. VSI HoloMedicine (VSI) broadly has 4 functions, which are surgical planning, patient education, teaching and clinical research.

Surgeons can upload 2D medical imaging files from their computer or picture archiving and communication systems (PACS), which is converted to 3D-rendered holographic model. These created 3D models can be visualised on HoloLens2 for surgical planning. Beyond simply viewing, 3D models can be also superimposed on patients in the theatre. VSI surgical plans were used in liver surgery, craniomaxillofacial surgery, neurosurgerie, ventricle drain placement, hepatobiliary pancreatic surgery, and parotid surgery in different studies.^10–13

Thorough understanding of the anatomical features and the relations are essential in medical education and surgical training. VSI offers different interactive tools that can be utilised with converted 3D models from 2D CT and MRI scans, providing in-depth understanding and visualisation to the students and the trainee doctors. Furthermore, natural and bone rendering functions of VSI offering photorealistic visualisation have potentials in increasing the understanding of the anatomy for medical students. Tools such as incision site marking, catheter placement assistance and streaming live surgeries could be beneficial in training trainee doctors.

VSI patient education offers surgeons or clinicians the ability to present 3D models of CT or MRI scans to explain the condition or interventions that patients are undergoing. This could support in-depth understanding for patients about the interventions they are undergoing.

While Holo-Medicine in general is being considered as a next generation medical technology, we aim to investigate the current use of VSI HoloMedicine (ApoQlar, Hamburg, Germany) in different medical specialities and discuss the potentials of its implementation in T&O.

Jain et al. assessed the usability of VSI as a neuronavi-gation system, recruiting three patients diagnosed with intracranial tumour requiring surgical resection.¹² Standard MRI scans were uploaded on the software and converted to 3D-rendered model. 3D model was superimposed on patient’s head via surface landmarks for the operations. The surgical incision and the craniotomy were guided based on an overlaying 3D model on the patients. Despite the visibility issues of rendered images due to the theatre light, all three surgeries were reported as uneventfully completed. Authors insisted that VSI showed advantages in such as accessibility, sterility and in-built slicer box tool. Operating surgeon expressed that the slicer box tool offering surgeons to observe inner part of the 3D model of the patient’s MRI scan was particularly effective for assessing tumours located anatomically deep.

Pérez et al. evaluated the intra-operative use of VSI and HoloLens2 for hepatobiliary-pancreatic surgeries and its use in surgical planning.¹³ Surgeries included 5 liver resections, 4 liver surgeries requiring resection and replacement of vena cava and 2 pancreatic procedures. Medical imaging (CT; MRI) converted to 3D-rendered models displayed on HoloLens2 were precisely examined and evaluated before the operations. All six participating surgeons reported that surgical preparations using VSI and HoloLens2 were easier compared to the conventional way of using 2D display. Regarding to their intra-operative use, these case series discussed 3 advantages in using VSI replacing the conventional method. First, the ability to visualise patient’s medical imaging in interactive 3D-rendered holographic models in the air during the surgery, maintaining the sterility. Second, the streaming function of VSI to share the procedure in real-time with consultants at other locations who can share their opinions or instructions back to the operating surgeon without physically attending to the theatre. Third, saving space in the theatre by replacing the physical computer monitors with HMD, which was HoloLens2 in this case.

Saadya et al. performed parotid surgeries in patients diagnosed with parotid gland neoplasm.¹¹ Obtained medical imaging were segmented in relative anatomical structures using different 3D slicer software. VSI and HoloLens2 were used for better visualisation and to interact with created 3D-rendered models. The 3D-rendered models of parotid glands and facial nerves were pre-assessed by the surgeons for the comparison with real-life samples and were shared with the patients preoperatively via HoloLens2. Following the comparison between 3D models viewed on HoloLens2 and the video review from physical surgery, facial nerves for all patients were successfully identified and accurately presented on 3D models. Advantages of using VSI that the article discussed were visualisation and interaction. Anatomical structures such as the buccal branch of facial nerve, which are difficult to identify and follow through were successfully depicted in 3D-rendered models that can be manipulated by the surgeon by rotating, re-sizing, and slicing. Furthermore, HoloLens2 conveying the 3D model to the surgeon not via 2D technology such as computer screen benefited in better understanding of patient’s anatomical features and their variances.

Fidan et al. used VSI for intra-operative navigation in Associating Liver Partition and Portal Vein Ligation for Staged Hepatectomy (ALPPS).¹⁰ 3D-rendered model of the liver was reconstructed using a software called “Visible Patient” (IRCAD, Starsbourg, France). The 3D model of the liver showing metastases and vascular structures in different colours were scaled at the surgical site, superimposed on the patient using VSI and HoloLens2 for navigating ALPPS. The author proposed that VSI was beneficial in localisation of the metastases particularly centrally located. As a result of using superimposed 3D holographic imageries, it significantly facilitated the preparation of central vessels in the liver hilum for parenchymal transection during first step of ALPPS.

Sun et al. described methods importing CT images of trauma patients to VSI for precise examinations of injury sites.¹⁴ 3D images created by the free-open-source 3D slicer software were imported to VSI to create interactive 3D-rendered models, viewed on HoloLens2. The performance of the VSI was quantitatively assessed by the questionnaire designed by the authors comparing with the traditional 2D images. Participants commented that VSI was beneficial in observing rib fracture, lung contusion, liver laceration, pelvic fracture, and cervical spine fracture with immersive 3D-rendered models, compared to scrolling up and down with CT images on 2D computer screen, using 2D DICOM viewer.

Theivendrampillai et al. conducted a single-centre study, performing trans-perineal prostate biopsy using VSI for prostate cancer diagnostics.¹⁵ 10 patients with a suspected prostate cancer were recruited and underwent 3T MRI with standardised sequences. Axial (T2-weighted) MRI scans were used for creating 3D-rendered models on VSI software. With HoloLens2, these models were projected on the users’ field of view. Pre-uploaded and highlighted regions of interest were examined by the surgeons before the biopsy. After confirmation, freehand PrecisionPoint trans-perineal biopsy was performed guided by the 3D-rendered models projected on HoloLens2 superimposed on the patients. After removing HoloLens2, standard biopsy was performed for comparison. The author concluded that the use of VSI and HoloLens2 for freehand PrecisionPoint trans-perineal biopsy is a feasible option.

House et al. conducted a prospective randomised clinical study investigating the superiority of using VSI over, using MRI on 2D computer screens or use of rubber-brain 3D model for explaining interventions and diseases to the patients.¹⁶ Participated patients were females and males of different age groups (21–61 years old) with different highest school-leaving qualification, who were scheduled to undergo 3 different surgeries (Epileptic surgery resections; stereotactic implantation of Anterior Nucleus of the Thalamus-Deep Brain Stimulation electrodes; stereotactic implantation of depth electrodes for stereo-electroenceph-alography). Following the randomisation, both groups were informed by the doctors by conventional method of educating patient. The additional educations were delivered with either VSI software or rubber brain model to each group. Patients and relatives completed the questionnaires assessing the effectiveness of the education method in three aspects (1. Get informed more comprehensibly, 2. Feel safer and less anxious about their upcoming surgery and 3. Feel treated by the state-of-the-art information technology). Significantly more patients reported that using 3D-rendered models with VSI was more comprehensible and imaginable. The results also showed that significantly more patients felt less anxious and reported VSI as a preferred tools to receive education about the surgery they are undergoing.

Skórka et al. conducted a pre-post intervention study investigating the effectiveness of using MR for educating patients who are having percutaneous balloon angioplasty (PTA) in the lower limb for treating peripheral artery disease (PAD).¹⁷ Included patients were in different age groups (≥18), diagnosed with chronic limb ischaemia IIb F by CT scanners. Personalised, 1:1 scaled 3D-rendered models were created based on the CT images taken from each patient. These models were then uploaded to the VSI software for presentation to the patients. Patients were then entitled to complete the questionnaire both pre- and postoperatively. The mean score of the knowledge test between pre- and postoperative questionnaires showed significant difference.

Ganeshkumar et al. investigated the difference between objective performances of neurosurgical residents and practicing neurosurgeons with VSI running on HoloLens2 and conventional DICOM viewer.¹⁸ 22 participants were randomised in two groups and CT images in DICOM format were obtained from 2 patients diagnosed with bony craniovertebral joint (CVJ) anomaly. These CT scans were uploaded on VSI to create and visualise 3D-rendered models on HoloLens2 for the participants. 4 stations were prepared, 2 with using HoloLens2 and 2 with using DICOM viewer to visualise CVJ anomalies of 2 patients. All participants from both groups rotated all 4 stations and recorded performance scores. The median performance score of both cases for both groups showed statistically significant improvement in stations using VSI and HoloLens2, compared to using DICOM viewer.

Along with evaluation of using VSI in hepatobiliary-pancreatic surgeries, Pérez et al. also conducted a satisfaction survey given to the medical students after participating three pilot classes using MR technology.¹³ The classes were real-time streaming of surgeries (open pancreatoduodenectomy; robotic hepatic segmentectomy; laparoscopic cholecystectomy) from the theatre combined with theoretical introduction. The survey consisted of 12 ques-tions, evaluated the technical quality and certain teaching elements. 83.3% expressed that content using MR technology facilitated the understanding of the surgery and 64.3% responded that it was easier to visualise and understand the surgical process with MR technology, compared to physically being in the theatre room. 95.2% of the students expressed that they would like to have this new method of teaching in the official teaching course.

Knee arthroplasty is a surgical procedure replacing the damaged (arthritic) knee joint with prosthetic components.¹⁹ Total knee arthroplasty (TKA) and uni-compartmental knee arthroplasty (UKA) are indicated in different patient groups,²⁰ with osteoarthritis being commonest in the United Kingdom.²¹ TKA benefits patients in pain relief, and improved functional mobility,²² and is considered as one of the most common, cost effective, and successful operations in the orthopaedics.²³ Despite its effectiveness and benefits, revision surgery raises concern worldwide and postoperative patient satisfaction varies. Infection, aseptic loosening and peri-prosthetic fracture are commonest three causes of revision TKA.²⁴ One systematic review investigated that most of the studies demonstrated patient satisfaction greater than 80%, yet the range of unsatisfied patient outcomes still spanned between 10%–20%, suggesting 1 in 5 patients are not satisfied with their outcomes following the TKA.²⁵

Preoperative planning of TKA plays crucial role in pa-tient outcomes, including range of motion (ROM) exami-nation, Collateral and Cruciate Ligaments assessment, neuro-vascular examination, radiographic evaluation and laboratory tests.²⁶ Different surgical techniques can be utilised when performing and balancing a TKA. One of the commonest methods used world-wide involves resecting specific amounts of the proximal tibia and distal femur in relation to their mechanical axis. The conventional way for these resections involves the use of an extramedullary tibial jig and a distal femoral intramedullary jig.²⁷ The distal femur and the proximal tibia are resected at specific angles, based on preoperative imaging and planning. The adjusted angle can vary amongst patients depend on their condition, lifestyles, and BMIs. Poor adjustments of the mechanical axis of the femur and the tibia can affect the patients’ weight bearing, which could result in revision surgery or failure.^1,27 Improper placement of the femoral and tibial components or component oversizing are one of the surgical risk factors that can increase the rate of post TKA failure. Driesman et al. showed an inaccuracy of using intramedullary jig for femoral resection guide.²⁸ To overcome these challenges, computer/robotic-assisted navigations were implemented. These navigating systems increased the precision, thus the success rate, yet it still poses some limitations.

The major difference between the RA-TKA and the conventional TKA is the navigation system of the operation. Currently, RA-TKA benefits in the accuracy of resection, minimal soft tissue injuries and patient specific mechanical axis adjustment.^29–31 However, one of the notable drawbacks of RA-TKA is that the medical imaging is delivered to the surgeons in 2D technology, which is a computer screen placed in the theatre.²⁹ Although RA-TKA provides surgeons a 3D image of patient’s tibia and femoral anatomical structures, these are displayed on 2D screen.

Possible points of implementation of VSI and HoloLens2 in TKA broadly include supporting preoperative planning and replacing intra-operative navigation system. Radiographic evaluation of anterior-posterior and sagittal views of the femur and tibia are essential in preoperative templating for TKA.³² Radiographic evaluation of long-leg views and the adjustment of posterior tibial slope are not always in routine assessment at every institution.³³ However, evidence shows that assessment of long-leg views is beneficial in preoperative templating^34,35 and that posterior tibial slope adjustment benefits in postoperative patient outcomes of TKA³⁶ and UKA.³⁷ With MR technology of VSI and displaying 3D-rendered holographic bone model on visual field of HoloLens2, surgeons can benefit in preoperative preparation of TKA in two ways. First, sur-geons could visualise the patient’s CT scans with an immersive 3D-rendered model with better spatial understanding of the anatomical features. Depends on the severity of bone deformities, it could be challenging to evaluate scans on 2D display. Although conventional DICOM viewer allow to view different plane views of 3D bone models, they are still delivered within the 2D computer screen by scrolling up and down. Sun et al. used a single interactive 3D-rendered model which surgeons can rotate and re-size with hand gestures in the air, offering more convenient radiographic evaluation whilst maintaining sterility, compared to the conventional radiographic evaluation using 2D DICOM viewer.¹⁴ Second, preoperative templating could be done directly on the 3D-rendered model using interactive drawing tools. With conventional methods, templating is usually done by placing templates over 2D images, drawing mechanical axis, anatomical axis, and lines perpendicular to the mechanical axis of each tibia and femur.³⁸ This is essential for deciding how much resection should be done on patients to perform optimal mechanical axis alignment. Alignment errors are critical in postoperative outcomes of TKA.^39,40 If the pre-operative alignment plans could be superimposed directly onto the 3D-rendered bone model in real time with immersive technology, it could potentially offer better patient-specific templating, intra-operative assessment of implant positioning, and thus lead to improved postoperative outcomes.

Intra-operative implementation of VSI and HoloLens2 in T&O is thought to be a potential replacement of Robotic-Assisted Navigation System. Despite the benefits of RA-TKA over traditional TKA, the computer monitor screen used for the navigation system displaying the patient’s models to the surgeons occupies a large space in the theatre.^41,42 Furthermore, they are financially more expensive compared to the conventional TKA. Utilising VSI and HoloLens2 intraoperatively in RA-TKA can benefit in 3 main aspects. First, surgeons would be able to use the 3D-rendered model for guidance throughout the operation rather than looking at the 2D screen. With superimposing function, it could be possibly used for guiding incision site as shown in Jain et al.¹² Second, replacement of the monitor screen used in RA-TKA system can save space. The monitor screen currently used in the theatre occupy significant space, which restricts movements in theatre and space for instrumentation. If the HoloLens2 could replace the motion capture cameras identifying the sensors attached on the tibia and femur, as well as project the 3D-rendered model, this could save significant space within theatre. Thirdly, HoloLens2 currently costs around £3000, whilst the RA-TKA systems can cost around £20,000.

There are numerous methods for obtaining tissue biop-sies.⁴³ Core biopsy utilises a needle to remove a small tissue sample. This can be image guided to increase accuracy up to 97% and minimise contamination.⁴⁴ Prior to the biopsy, radiological evaluation of the region of interest is essential to decide entry point, type of biopsy and safest route to avoid contamination. The feasibility of using VSI and HoloLens2 in tumour resections and as an alternative biopsy technique are already shown.^12,15 Using 3D-rendered models created by VSI for image-guided musculoskeletal biopsy can enhance evaluation, preparation, and navigation. First, the position of the biopsy site could be complex due to the regional morphologic variations of musculoskeletal tumours.⁴³ Using superimposed 3D-rendered models highlighting the suspected tumours of interest on the patient could enhance the preparation by making decision on biopsy technique and approach easier as Theivendrampillai et al. showed.¹⁵ Second, 3D-rendered models projected on HMD showing lesions in different colours can benefit surgeons to understand the location and anatomical relations of musculoskeletal tumours more conveniently, thus bene-fiting in evaluating the location of suspected tumours.^10,12 Moreover, considering that the cortex of the long bone is one of the commonest sites of musculoskeletal tumour,⁴⁵ slicer toolbox of VSI visualising deep structures could be effective in evaluating and observing the suspected lesions or tumours which are located deep. Lastly, superimposing 3D-rendered model on patients can potentially replace or be utilised along with the imaging-guided navigation system intra-operatively.

A workforce shortage is a major worldwide rising issue in healthcare sector.⁴⁶ NHS is also under a huge burden of staff shortages,⁴⁷ financial issues and long waiting times for the patients. To address this, national regimes include increases in medical student admissions and establishment of new medical schools.⁴⁸ Major issues with this are lack of spaces and opportunities for education and training. MR has potentials in supplementary adjuvant to the medical education and surgical training.

T&O particularly emphasises the thorough understanding of the anatomical features and relations of the bony structures and soft tissues. Cadaveric, plastinated or plastic models are suggested to have superiority in anatomy education over 2D images in the textbook.^49–51 However, both cadavers and plastinated specimens from real human samples require laboratory space. These laboratories have limited numbers of students they can hold and are restricted with infection regulations. Immersive platforms created by VSI, presenting 3D-rendered anatomical models on HMD could possibly lower the burden of lack of laboratory space with increasing number of students. Apart from the spatial issues, it could potentially lower the financial burden by replacing laboratories with several HMD. However, HMD and VSI cannot be solely used for medical education by themselves. Sociocultural theory proposed by psychologist Vygotsky has proposed that people have the greatest learning curve by socio-culturally interacting with each other, when constructing knowledge.⁵² Considering the significance of socio-cultural interaction in learning, educators using HoloLens2 together with the students, combined with explanation and teaching would be more suitable then completely replacing laboratories with HMD. Furthermore, real human samples as medical and anatomical education are still proven to have numerous advantages, thus VSI would be helpful as a supplementary teaching resource.

Surgical training in T&O could benefit from VSI and HoloLens2 in 3 different aspects. First, streaming can offer trainee doctors the opportunity to observe procedures in first-surgeon view. As trainees could remotely view the feed projected from one HMD worn by an operating surgeon, this would lead to greater training opportunities for a larger cohort of trainees, as well as free up space in the theatre. These streaming functions could be implemented in medical school teaching as a supplementary resource for lectures. Second, MR platforms offering a 3D-rendered models which trainee doctors could interact with can be effective for learning complex surgeries. Due to its intrinsic nature, T&O requires using various surgical tools, robotics-navigated systems. Creating a 3D-rendered models of the patients and virtual stations with VSI to practice and learn procedural steps of operations such as TKA, Osteotomy, Musculoskeletal biopsy could yield better training experi-ence for doctors. Thirdly, surgeons could be remotely as-sisted from individuals in other institutions. As shown in Pérez et al., other surgeons or expert from corporates do not need to be physically at the theatre, which resolve spatial, transportation and infectious regulation-related issues.¹³

Kolécki et al. assessed the utility of MR in medical education by conducting a survey using the Lickert scale.⁵³ 211 medical students and 47 academic faculty at the medical school participated the survey. Result showed that 66.9% and 63.9% of medical students and faculty respec-tively agreed to the questionnaire stating, ‘Education using AR/VR technology has more advantages than classic education’. Furthermore 84.8% and 72.3% of medical students and the faculty respectively believed that the use of AR/VR technology increases educational opportunities and improves participation in classes. This reinforces the potentials of using VSI in medical education and surgical training for T&O.

Radiographic evaluation in T&O is one of the most significant preoperative assessments. It is essential to both the surgeon and patient to understand their condition and the proposed plan. Radiological assessment and expla-nations can be challenging to patients, especially when assessing on a 2D image. House et al. showed, using photorealistic 3D-rendered models of the medical images, trauma cases could be presented to the patients more easily.¹⁶ This could eventually result in better understanding of the patient, and thus help in patient’s decision making and consent. House et al. and Skórka et al. demonstrated higher comprehension scores of patients after using VSI for patient education, supporting the use of these methods in patient education/consent.^16,17

Although reviewed studies presented the positive feasi-bility of implementing VSI in different medical specialities, technological improvements are still necessary to achieve some of the points discussed in this review to be fully integrated into T&O practice. For example, steadily locking the superimposing holograms on the patients still require technical improvement. This reinforces the necessity of designing and conducting future studies that can evaluate the realistic challenges such as long-term effect of VSI, cost effectiveness, needs of special training and integration with existing medical techniques.