PhD opportunity on "Joint learning of stereo vision reconstruction and hyperspectral imaging upsampling for binocular surgical guidance"

Project overview:

• Title: Joint learning of stereo vision reconstruction and hyperspectral imaging upsampling for binocular surgical guidance
• First supervisor: Tom Vercauteren
• Second supervisor: Christos Bergeles
• Clinical Supervisor: Jonathan Shapey
• Start date: October 2022

Project summary

Optimal outcomes in oncology surgery are hindered by the difficulty of differentiating between tumour and surrounding tissues during surgery. Real-time hyperspectral imaging (HSI) provides rich high-dimensional intraoperative information that has the potential to significantly improve tissue characterisation and thus benefit patient outcomes. Yet taking full advantage of HSI data in clinical indication performed under binocular guidance (e.g. microsurgery and robotic surgery) poses several methodological challenges which this project aims to address. Real-time HSI sensors are limited in the spatial resolution they can capture. This further impacts the usefulness of such HSI sensors in multi-view capture settings. In this project, we will take advantage of a stereo-vision combination with a high-resolution RGB viewpoint and a HSI viewpoint. The student will develop bespoke learning-based computational approaches to reconstruct high-quality 3D scenes combining the intuitiveness of RGB guidance and the rich semantic information extracted from HSI.

Project plan

The student will design novel computational approaches to optimally embed hyperspectral imaging in binocular surgical vison, thereby providing novel tissue characterisation capabilities. Real-time mosaic sensor based hyperspectral imaging (HSI) is a camera-based optical imaging technique that split light into multiple narrow spectral bands across the pixels of the sensors. It enables the acquisition of much richer information than the RGB information that can be seen with the naked eye or standard cameras but can only be done in real-time with a restricted spatial resolution. It also faces additional challenges related to the unknown geometry of the scene which hinders some of the quantitative spectral measurement capabilities. The project plan aims at addressing these challenges. We hereby concisely list the deliverables of the PhD and associated timelines.

Year 1

• Set up of a first stereo vision rig with a high-resolution(4k) RGB camera and a mosaic sensor (1080p across channels) HSI camera separated by a baseline
• Geometric calibration of the imaging system
• Acquisition software to capture synchronised data stream
• Implementation and validation of learning-based dense stereo RGB reconstruction using the high-resolution RGB combined with RGB-from-HSI (as reconstructed using a state-of-the-art standalone HSI super-resolution pipeline)
• Phantom creation for validation purposes

Year 2

• Refinement of the acquisition rig
• Development of RGB-guided HSI super-resolution approaches to achieve 4K HSI spatial reconstruction (factor 2 beyond the state of the art)
• Implementation of the RGB-guided upsampling to improve dense depth reconstruction
• Publication

Year 3

• Algorithm development to use the recovered depth information to achieve better quantitative spectral measurement
• Software stack optimisation and development of end-to-end approaches to achieve (near) real-time reconstruction and spectral calibration
• Publication

Year 4

• Embedding of HSI-derived semantic segmentation and tissue features (e.g. perfusion) in the binocular reconstruction
• Phantom refinement
• Thesis write-up and publication

More information about the PhD project and how to apply on the website of the Centre for Doctoral Training in Surgical & Interventional Engineering.