Haptics and its Applications to Medicine

Haptics, computer interface technology that mediates the sense of touch, is a rapidly growing field with applications in areas ranging from tele-robotics, entertainment, and education to realistic simulators for training and planning of complex surgical procedures. The Centre for Image Analysis has conducted haptics research for more than a decade, with research in hardware for multi-finger haptics, and software that includes haptics-assisted assembly of fractured objects, interactive segmentation of medical images, and cranio-maxillofacial (CMF) surgery planning.

Mandibular Reconstruction Planning with the Haptics-Assisted Surgery Planning (HASP) System

HASP, our Haptics-Assisted Surgery Planning System is based on the research described below and on fast, interactive segmentation. We demonstrate a planning session in the HASP system of a mandibular (lower jaw) reconstruction using bone segments grafted from the patient's fibula. A surgeon may interactively and iteratively design an optimal reconstruction by defining and refining fibula osteotomies, anastomosis sites, and configuration of a skin paddle to cover soft tissue deficits.

Custom Mandibular Implant Design with Deformable Models and Haptics

We propose a semi-automatic method based on deformable models and haptics that allows a surgeon to easily design, adjust, and virtually test the fit of a scaffold implant before production.

SplineGrip - An 8 DOF Flexible Haptic Sculpting Tool

SplineGrip is a flexible haptic sculpting tool that senses the articulation of the hand in two degrees-of-freedom (DOF). The tool is mounted on a commercial haptic device that tracks hand pose (position and orientation in six DOF) while simultaneously providing three DOF haptic feedback to the hand.

Haptics-Assisted Cranio-Maxillofacial Surgery Planning

We are developing a system for planning the restoration of skeletal anatomy in facial trauma patients using a virtual model derived from patient-specific CT data. The system combines stereo visualization with six degrees-of-freedom, high fidelity haptic feedback that enables analysis, planning, and preoperative testing of alternative solutions for restoring bone fragments to their proper positions.

Snap-to-fit, a Haptic 6 DOF Alignment Tool for Virtual Assembly

Precise fitting of fractured objects guided by delicate haptic cues similar to those present in the physical world requires haptic display transparency beyond the capabilites of today's systems. We have developed a haptic alignment tool that combines a six degrees-of-freedom (DOF) attraction force with traditional six DOF contact forces to pull a virtual object towards a local stable fit with a fixed object.

Perceptual Evaluation of Co-located and Non-Colocated Haptics

Co-located haptics will gain importance when more advanced haptic interfaces, such as high-fidelity whole hand devices, become available. We have investigated the pros and cons with physically co-located versus non-collocated haptics on stereoscopic displays.

Whole Hand Haptics with True 3D Displays

Our long term goal is a new interaction paradigm that gives the user an unprecedented experience to touch and manipulate high contrast, high resolution, three-dimensional (3D) virtual objects suspended in space, using a glove that gives such realistic whole hand haptic feedback that the interaction closely resembles interaction with real objects using a bare hand.

Orbit Segmentation

One important component in surgery planning is to accurately measure of the extent of certain anatomical structures, such as the shape and volume of the orbits (eye sockets). These properties can be measured in 3D computed tomography (CT) images of the skull, which requires the segmentation of the orbits from the rest of the image. In this project, we are developing a semiautomatic system for segmenting the orbit.

Feeling is Believing

We have conducted studies on how to generate tactile stimulation to the fingertip.

Visualization and Haptics for Interactive Medical Image Analysis

The WISH toolkit comprises algorithms and methods for interactive medical image analysis using volume visualization and haptics. The toolkit is licensed under the GNU public license (GPL). The core of WISH is a stand-alone C++ library with implementations of image analysis algorithms, visualization algorithms, and haptic rendering algorithms, integrated with the H3D API from SenseGraphics AB.

People

Publications

Peer-Reviewed Journal and Conference Articles

  1. P. Olsson, F. Nysjö, I.B. Carlbom, and S. Johansson, "Comparison of Walking and Traveling-Wave Piezoelectric Motors as Actuators in Kinesthetic Haptic Devices", To appear in IEEE Trans. Haptics, 2016.
  2. P. Olsson, F. Nysjö, N. Singh, A. Thor, I.B. Carlbom, "Visuohaptic Bone Saw Simulator Combining Vibrotactile and Kinesthetic Feedback", ACM SIGGRAPH Asia, Kobe, Japan, November 2015.
  3. P. Olsson, "Haptics with Applications to Cranio-Maxillofacial Surgery Planning", Ph.D. Thesis, 2015. (PDF)
  4. Johan Nysjö, Filip Malmberg, Ida-Maria Sintorn, Ingela Nyström, "BoneSplit - A 3D Texture Painting Tool for Interactive Bone Separation in CT Images," Journal of WSCG, 2015.
  5. I. Nyström, P. Olsson, J. Nysjö, F. Nysjö, F. Malmberg, S. Seipel, J.M. Hirsch, I.B. Carlbom, "Virtual Cranio-Maxillofacial Surgery Planning with Stereo Graphics and Haptics", Book Chapter, Computer-Assisted Musculoskeletal Surgery, Springer, Switzerland, 2015.
  6. P. Olsson, F. Nysjö, A. Rodriguez-Lorenzo, A. Thor, J.M. Hirsch, and I.B. Carlbom, "Novel Virtual Planning of Bone, Soft-tissue, and Vessels in Fibula Osteocutaneous Free Flaps with the Haptics-Assisted Surgery Planning (HASP) System", to appear in Plastic and Reconstructive Surgery Global Open (PRSGO), 2015.
  7. F. Nysjö, P. Olsson, J. Hirsch, and I.B. Carlbom, "Custom Mandibular Implant Design with Deformable Models and Haptics", Proc. Computer Assisted Radiology and Surgery (CARS), Fukuoka, Japan, June 2014.
  8. P. Olsson, F. Nysjö, B. Aneer, S. Seipel, and I.B. Carlbom, "SplineGrip - An Eight Degrees-of-Freedom Flexible Haptic Sculpting Tool," ACM SIGGRAPH 2013, Anaheim, USA, 2013 (Abstract).
  9. P. Olsson, F. Nysjö, J.M. Hirsch, and I.B. Carlbom, "A Haptics-Assisted Cranio-Maxillofacial Surgery Planning System for Restoring Skeletal Anatomy in Complex Trauma Cases," International Journal of Computer Assisted Radiology and Surgery (IJCARS), 2013. (PDF)
  10. P. Olsson, F. Nysjö, J.M. Hirsch, and I.B. Carlbom, "A Haptics-Assisted Cranio-Maxillofacial Surgery Planning System for Restoring Skeletal Anatomy in Complex Trauma Cases," Proc. Computer Assisted Radiology and Surgery (CARS), Heidelberg, Germany, June 2013.
  11. P. Olsson, F. Nysjö, J.M. Hirsch, and I.B. Carlbom, "Snap-to-Fit, a Haptic 6 DOF Alignment Tool for Virtual Assembly," IEEE World Haptics Conference, Daejeon, South Korea, 2013. (PDF)
  12. P. Olsson, S. Johansson, F. Nysjö, and I.B. Carlbom, "Rendering Stiffness with a Prototype Haptic Glove Actuated by an Integrated Piezoelectric Motor," Euro Haptics Conference, Tampere, Finland, 2012. (PDF)
  13. P. Olsson, F. Nysjö, S. Seipel, and I.B. Carlbom, "Physically Co-Located Haptic Interaction with 3D Displays," Haptics Symposium, Vancouver, Canada, 2012. (PDF)
  14. L. Svensson, J. Nysjö, A. Brun, I. Nyström, I-M. Sintorn, "Rigid Template Registration in MET Images Using CUDA," in International Conference on Computer Vision Theory and Applications (VISAPP 2012), Rome, Italy, 2012. Volume 2, pp 418-422.
  15. J. Nysjö, A. Christersson, F. Malmberg, I-M. Sintorn, I. Nyström, "Towards User-Guided Quantitative Evaluation of Wrist Fractures in CT Images," in International Conference on Computer Vision and Graphics (ICCVG 2012), Warsaw, Poland, Springer-Verlag; 2011. p. 204-211. Lecture Notes in Computer Science, 7594. (PDF)
  16. I. Nyström, J. Nysjö, F. Malmberg, "Visualization and Haptics for Interactive Medical Image Analysis: Image Segmentation in Cranio-Maxillofacial Surgery Planning," in: Visual Informatics: Sustaining Research and Innovations. Berlin Heidelberg: Springer-Verlag; 2011. p. 1-12. Lecture Notes in Computer Science, 7066. (PDF)
  17. L. Svensson, I. Nyström, S. Svensson, I-M. Sintorn, "Investigating measures for transfer function generation for visualization of MET biomedical data," in Proceedings of WSCG '2011. Plzen, Czech Republic: Union Agency; 2011. p. 113-120.
  18. F. Malmberg, I. Nyström, A. Mehnert, C. Engstrom and E. Bengtsson, "Relaxed Image Foresting Transforms for Interactive Volume Image Segmentation," in Proceedings of SPIE Medical Imaging, 2010.
  19. I. Nyström, F. Malmberg, E. Vidholm, and E. Bengtsson, "Segmentation and Visualization of 3D Medical Images through Haptic Rendering," in V. Krasnoproshin, S. Ablameyko, R. Sadykhov, eds., Proceedings of the 10th International Conference on Pattern Recognition and Information Processing (PRIP 2009), pages 43-48. Publishing Center of BSU, Minsk, Belarus, 2009.
  20. E. Vidholm, M. Golubovic, S. Nilsson, and I. Nyström, "Accurate and reproducible semi-automatic liver segmentation using haptic interaction," in Michael Miga, Kevin R. Cleary, editors, Medical Imaging 2008: Visualization, Image-Guided Procedures, and Modeling, Proc. SPIE 6918, 2008.
  21. E. Vidholm and I. Nyström, "Haptic interaction with deformable models for 3D liver segmentation," In Proceedings of MICCAI Workshop: Interaction in Medical Image Analysis and Visualization, Brisbane, 2007.
  22. F. Malmberg, E. Vidholm, and I. Nyström, "A 3D live-wire segmentation method for volume images using haptic interaction," in Attila Kuba, editor, Proceedings of Discrete Geometry for Computer Imagery (DGCI 2006), volume 4245 of Lecture Notes in Computer Science, pages 663-673. Springer-Verlag, 2006.
  23. E. Vidholm, S. Nilsson, and I. Nyström, "Fast and robust semi-automatic liver segmentation with haptic interaction," in Rasmus Larsen, editor, Proceedings of Medical Image Computing and Computer-Assisted Intervention (MICCAI 2006), volume 4191 of Lecture Notes in Computer Science, pages 774-781. Springer-Verlag, 2006.
  24. E. Vidholm and I. Nyström, "A haptic interaction technique for volume images based on gradient diffusion," in IEEE Proceedings of WorldHaptics 2005, 2005.

Other Publications

  1. I.B. Carlbom and P. Olsson, "Surgical Training Using a Haptic-Assisted Cranio-Maxillofacial Planning System", CIG on Simulation Education Program, 97th Annual meeting of AAOMS (American Association of Oral and Maxillofacial Surgeons), Washington, DC, USA. October 2015 (Oral presentation).
  2. I.B. Carlbom, E. Bengtsson, "Report on Whole Hand Haptics with True 3D Displays, research project in the KK-Stiftelsen, VINNOVA, SSF, ISA and Vårdalstiftelse Visualization Program", diarie number 2009/0087, final report, 2012.
  3. J.M. Hirsch, K. Gamstedt, J. Sörensen, L.E. Rännar, I.B. Carlbom, "Virtual Planning of Reconstructions, Production of Individualized Implants, and Transfer of the Plan to the Operating Room in CMF Surgery," Materialise World Congress, Leuven Belgium, April 2012 (Abstract).
  4. J.M. Hirsch, K. Gamstedt, J. Sörensen, L.E. Rännar, I.B. Carlbom, "Virtual planning of reconstructions and transfer to the operating room in CMF surgery," 26th World Congress of the International College of Maxillofacial Surgery, Gran Canaria, Spain, February 2012 (Abstract).
  5. M. Nilsson, "Evaluating the Accuracy of Motion Tracking Algorithms for Determining the Position of Finger Tips", MSc thesis, Department of Information Technology, Uppsala University, 2012. (PDF)
  6. P. Olsson, S. Johansson, and I.B. Carlbom, "Whole Hand Haptics," Medicinteknikdagarna 2011, Linköping, Sweden, pp. 98, October 2011 (Abstract).
  7. F. Nysjö, "Rendering Software for Multiple Projectors," MSc thesis, Department of Information Technology, Uppsala University, 2011. (PDF)
  8. J. Björk, "Evaluation of a Holographic 3D Display," MSc thesis, Department of Information Technology, Uppsala University, 2010. (PDF)
  9. J. Nysjö, "Orbit Segmentation for Cranio-Maxillofacial Surgery Planning," MSc thesis, Department of Information Technology, Uppsala University, 2010. (PDF)

Popular Science Articles

  1. "Virtuell kirurgi," Tandläkartidningen, issue 6/2013, pp. 12-15, 2013.
  2. E. Vidholm, F. Malmberg, and I. Nyström, "Läkarens känsel hjälper datorn" Forskning & Framsteg, 1:62, 2009.