Summary of Medical Research - Video

I have compiled a video to summarize the work done over summer. I hope it is helpful to everyone.

Presentation under construction

I am making a presentation for the project. I have made the slides; I am currently working on adding voice to the presentation.
For the presentation, please refer to:
https://docs.google.com/presentation/d/1w8bTrBoPRzjik5Nfkr136YjJ8bo2K5W54tyugfEmUmg/edit

Prototype

Ultrasound Guided Needle Placement Training Using Kinect

The prototype is complete!
The rendering module and the tracking module are fully integrated now. The needle visualization is also working. Right now the needle's radius (its actually a cylinder), is rather large. I did this in order to easily get the needle in screen, otherwise it would be very difficult. But this parameter is very easy to change, and can be tuned as per the needs of the ultrasound guided needle placement training.

Challenges:
Many of the challenges that we are facing are mechanical in nature. The probe and needle tracking is working well despite the fact we are not using any fancy setup for tracking them.

What has been found?
At the start of the project we were very anxious about the accuracy of the Kinect for tracking probe and needle. But we can now see that Kinect works very well for this application.

Future Work:
1. Improving accuracy of tracking by using sophisticated marker tracking setup
2. Developing the software in the form of a game so that new surgeons can monitor their own skill level

Long Due Blog Update - Rendering Ultrasound Images based on Kinect Input

Rendering based on Unmodified Kinect Input, discarding normal and up vectors for probe

Rendering Based on Stabilized Kinect Input, discarding Up and Normal Vectors from Kinect

Rendering Based on Stabilized Kinect Input, using Up and Normal Vectors from Kinect

Due to a number of reasons, I could not get a chance to update my blog. This was my progress as of August 3, 2012, just before SIGGRAPH conference.

I had completed integration of rendering module with the Kinect Tracking module.

The first video shows rendering based on data obtained from Kinect without any modifications; as you can see, the output shakes a lot which is not desirable.

To remove the shaking, I followed the simple approach recommended by Dr. Norman Badler, which was to just ignore some of the data given by Kinect. The second video and third video show outputs obtained using this approach. To improve the quality of the output Dr. Badler had suggested using B-Spline to interpolate between frames. I will do it in the future.

The second video discards the Up vector and Normal vector of the slice plane obtained using Kinect, which adds to the stability; but this method will prevent the doctors from being able to orient the probe along oblique directions. So this method cannot be used.

The third video shows the result of the integrated Tracking and Rendering module, without the needle.

All Spacial Information Collected for Both Probe and Needle

The above video shows the effect that all of the markers of the probe and the needle are being tracked by the camera.

The user should wear white coat and also white gloves if necessary, because skin color and the needle marker colors are similar thus confusing the camera. We will try to modify the color alignment to make it more convenient, but this might also be reasonable for trainees. The background is also better to be white.

As we noticed before, the data collected vibrates a lot. We are not sure how this would be like in the mock ultrasound output since we haven't integrated our codes. However, we are also trying to use some filter( Karman Filter or simpler, low pass filters) for optimization.

Cheers!

Multiple Updates - Needle Rendering/Integration

Needle Visualization

Now we have the needle visualization along with the medical data rendering. In the video I am moving the slicing plane front and back, so the needle is visible sometimes (when the slicing plane and the needle are nearly in the same plane). As the slicing plane gets near the needle plane the needle visualization becomes brighter and brighter. Also note that the slicing algorithm is capable of changing the needle visualization as the needle penetrates into or off the body.

Integration Methodology

Robin and I have very specific settings for each of our modules, so we decided not to integrate the modules at the code level. So I thought it would be good if we make Robin's module send the data that it captures using Kinect to my module using Shared Memory. After a little searching in the internet, I got a surprisingly easy method to implement shared memory.

In the video above, I created a dummy process, "Data Provider" which just randomly sends data regarding the probe position and orientation. The rendering module is rendering as per the data that it gets from the dummy process.

Now we are going to add the small piece of code in Robin's module so that it can send data to my module.

Probe 3D Spacial information

Got 3D Spacial information for all of the markers and the bottom surface of the probe.

The values vibrated heavily because all of the inputs are unstable. We are now trying to integrate our code to see the final results.

In the video the bottom line shows the probe surface position.

Getting Probe Information

Now we are able to identify all the markers and the mock dummy. With the markers' 3D spatial information we are trying to get the position of the probe first. We measure the distance between the center of the markers and get:


blue-green: 116mm
blue-yellow: 83mm

We define a black circle which is the upperleft corner of the mock dummy and will be the relative zero point in the xyz cordinates. x axis points towards right, y points up and z points out the image.
We also require the three markers of the probe be positioned at the rectangles drawn in the image to identify. The board which carries the blue and green markers should be horizontal.

From the given data we want to find the relation between the pixel index and the 3D coordinates information, which includes 1)Up vector; 2) View direction for the ultrasound image

Probe Simulator

After we confirmed that we could get accurate depth data from Kinect, we attached three markers to the mock probe and tracked the 3D spacial information of them. Based on this information, we will be able to detect the position, direction and rotation of the probe.

Since the markers are smaller than the cups that we used previously, to make sure they are sensitive enough, their contrast against the background should be as high as possible. So we used Tianyu's black coat and it works very well. Also the board which carries the markers is dark yellow, similar to skin, so we moved the stand instead of the probe itself which would confuse the camera between our skin and the board. However, the light yellow marker could be detected even if we hold the probe in hand. So it could be the same convenient for the trainee.

We will modify the color of the board, record the requirement of the background and create a mock dummy with a box to test the 3D spacial data for the probe. After that, we will also try to integrate our codes.

Depth Data Detected

Using the method in the last blog, we tested the depth value of the objects. We displayed the values on the output images and as seen in the above video, they are pretty accurate and quick.

There are some occasions that the values are zero or 17** which is the depth of the background wall. The first error is because the shadow of the cup moves its mass center from its real center to the bottom, which is very like to be "invalid area"; the second error happens when the object is moved very fast, because depth image falls behind the color image by one to two frames and actually returns the depth value of the same coordinates in the previous frame.

To solve the first problem, we just need to cover the table with a low reflective cloth to remove the shadows; the second one could rarely happen in practice, but I will also filter it out.

Calibrate Depth Map and Color Map

There is some offset between the color image and depth map. In order to get the depth data, we got some sample images and calibrated them.

Color Image

Depth Image

Calibration

So In order to get the corresponding pixel in the depth image from the color image, we should firstly clip the color image out of the rectangle (52, 30) (626,460), then amplify the coordinates (x, y) by (1/0.9) to get the coordinates(x', y') in the depth image.

Track Multiple Objects

We are now using the CamShift algorithm from OpenCV library to track objects. This algorithm identifies one object from the color of the selected area and track it using the histogram. We upgraded it to track multiple objects in the scene. It works well and could also solve hidden situations. But since it is sensitive to color, we have to distinguish the objects from background and themselves. Now we are going to:

1) test the algorithm with up to five objects

2)design the markers and calibration process.

3)gain information from depth map

Below is the demo for two objects tracking

Results of Slicing Volumetric Data Constructed from Real Ultrasound Images

Slices of Volumetric Data constructed from Real Ultrasound Image

Finally, we can see some images which are very close to the target of this project. I am moving the slice plane using keyboard right now. But soon enough Robin and I will start working on integrating this Rendering module with the Kinect Tracking module.

Optimized Tracking Algorithm

The above video shows the effect of the new tracking approach. This method is tracking the mass center of each object frame by frame. It is more accurate and faster, and it could return the mass center of each object, which is helpful to get the depth data from kinect's depth map. However, it cannot solve the hidden problem. And currently it cannot be run in release mode. Trying to figure it out. Also discussing with others about the possibility of using existing algorithms like Camshift.

Slicing Test Cases and Rendering in OpenGL

Slices of Cylinder Along Y(-Z) axis

Rendering Slices of Cylinder Along Z axis in OpenGL in Real Time

In the last update I had the slices of spheres and cloud. This time to really test the slicing algorithm, I used a cylinder and sliced it along many different directions. Along Z axis, there are rectangular slices. Along Y axis, there are circular slices. Along Y(-Z) there are elliptical slices. Thus, the slicer passed all the tests.

Further, I have used OpenGL for rendering the results dynamically. First I tested with a 100 x 150 image. It was running real time at that resolution. But then I increased the resolution to 510 x 335, at which the slicer became really "not real time".

Thankfully, I had spent time in the earlier stages of the project working on multi threading. I added multi threading to my slicing algorithm and now it runs in real time!

Multiple Object Tracking

We wrote a method to track multiple objects in the view scene. In the movie below, two different objects are identified as blue and black markers( from their original positions). Then they are tracked as being moved. However, the tracker sometimes has errors, for example, they can sometimes misidentify the blue one with the black one. The frame rate is also kinda low. Improving them. Trying to optimize the algorithm by estimating and tracking the mass center of each object.

Tracking with Original Calibration

We would like to track the different markers via identifying them at the original place. Thus we fixed the camera and the different objects as the image shows below. The original position of the markers are shown in the 2D camera image as:
GREEN:(187,266),(205,315)
RED:(463,292),(505,310)

The two coordinates are for the upper left corner and lower right corner of the two rectangles. We will use a matrix called identifier to define and track the objects in the view scene.

Sensitive to light

This morning the code worked pretty well, then in the afternoon it didn't. After a while I found out that the difference is that the light was turned on(see the circled area in the image below). So the corner detector is pretty sensitive to light in the scene, especially the array of lights which also form some kind of corner pattern. Take this note for further calibrations.

Marker Detector

Modified the marker to the Chessboard pattern.

Append the Marker to a pen so the cylindrical pattern could be seen from every direction.

The result is decent, as seen from the two images below.

Within the distance that is reasonable for the accuracy of Kinect Depth Sensor, both of the pens could be detected from the scene. More patterns would be added like color to identify between two objects.

More details will be shown later today or this weekend.

Kinect Depth Map Calibrated

Calibrated the Kinect Depth Sensor.

Picked the depth value of the mid point of depth map. Measured with A4 paper whose length is 279mm.
Data collected as below:

1674:A4*6-1661
1395:A4*5-1382
1116:A4*4-1111
837:A4*3-836
558-A4*2-NULL


In each row the first number is the actual length of the string of paper; 6/5/4/3 is the number of paper; the last number after "-" is the value is abstracted from Kinect Depth Sensor's return value. The value returned should right switch 3 digits(these 3 digits contain the index information).


So the sensor works pretty well in detecting the distance from the object. Especially between 800 to 1100, the error is limited within 5mm. That is the best area for Kinect to detect.

Multiple Updates

The videos above show slices obtained by varying the location of slicing plane. The first one has a sphere in the grid space, whereas the other one has cloud

After playing around with the open source volume renderer for a while, I realized that it would be better to make my own volume visualizer/slicer. I am able to obtain slices at a very rapid rate. Once I place these images in the OpenGL buffer, I think interactivity can be achieved easily. My estimate of the frame rate that can be obtained is 10 fps. But I am going to multithread this process, so I hope to achieve 30 fps. If that does not work then I will use CUDA to accelerate it.

I have already spent some time to get multi threading to work in my CIS 560 volumetric renderer. I am proud to say that the render time has come down from ~13 seconds to ~2.5 seconds. So I feel slicing which takes very little time as it is would take even lesser time after multithreading.

But I am curious as to how fast the whole system would be. So, once I get progress in other modules, I will start on this module again and use CUDA to offload the CPU off the rendering task.

Depth

The demonstration above shows that we are able to combine the information of the depth map with the color video stream. And render in 3D environment.

The OpenGL 3.1 is now included into the project. And it has become an integration of the OpenGL (3D display / calculation), OpenCV ( 2D image processing / marker tracking ), and Kinect API ( camera / depth map driver ).

The color marker tracking function mentioned in the last post is refactored to support arbitrary number of the marker. But the stability of our naive tracker is not ideal. And someone suggested us to look for the "Blob tracker" works done by other people. The idea is similar, but they have more filter and estimation for stabilizing the tracking process. cvblob is the one we currently intend to use, since it is built upon OpenCV we have, and works pretty well in the demonstration.

Right now we have all three dimension information in the space we need to read the operation of the trainee. But the error is still large, we hope that by adapting "Blob tracker", the output would be acceptable.

Quick update in the motion tracking

We successfully drives Kinect and retrieve the Video as well as Depth information from the sensor on the Kinect. The information is also successfully integrated with the OpenCV image processing library. So currently we have full access to all the functionality we want on the Kinect.

After discussion with Dr. Badler, we got a conclusion that the flat ARTag( http://www.artag.net/ ) mentioned in the project proposal is not an ideal tracking measure in the training environment. Though it provides accurate tracking feature, it can only function properly when the tag is facing the camera or within certain small angle. But the trainee might need to rotate the sensor or the needle during the training. 

Based on the conclusion, we decided to switch to the color tracking method. The screen capture above shows that we are able to track certain color in the environment. We are able to get the depth information at the corresponding position and get the 3rd dimension information, too. The next step would be tracking 2 or more markers to get the vector between them and try to align this vector to the sensor and needle we want to track.

Project Summary

Team: Chun Fu Chao and Aparajith Sairam

Advisor: Dr. Norman I. Badler

Based on:

An augmented reality simulator for ultrasound guided needle placement training

Authors: D. Magee. Y. Zhu. R. Ratnalingam . P. Gardner . D. Kessel

        We would like to develop an inexpensive and adaptive ultrasound guided needle placement training system that utilizes the advantage of Microsoft Kinect as a low-cost and  highly available motion tracking hardware.

In the conventional approach, needle placement training is done by foam model with plastic tube sealed within. But there are several drawbacks including: high unit cost ( around 2,000 USD for limb and over 25,000 USD for Full Torso [1]), limited use per model unit, and unrealistic scanning result compared to the human tissue. In 2011, Andrew D. Rosenberg, MD et al. proposed a fully computer simulated approach for ultrasound guided needle placement training. They collected the 2D ultrasound image planes from real human test subject and compiled images into the 3D voxel data. They also used magnetic motion tracking system to track the sensor and needle held by the trainees. Then a synthesized ultrasound image plane with fictionally injected needle is drawn on screen.

This paper opened the possibility of more realistic training environment aided by the power of modern computer. But the magnetic motion tracking system used by the paper author is still not an inexpensive option. The system set up requires two magnetic signal transmitter and two pairs of magnetic markers, which would summed up to around 7,000 USD [2]. The magnetic tracking system also requires user to disassemble the computer and install PCI cards onto motherboard, which will result in difficulties in installation and maintenance for the users who don’t have any computer hardware expertise.

So we would like to propose a new system that is solely based on Microsoft Kinect technology. Kinect Windows applications is now supported by the Microsoft development team, the new Application Programming Interface and new hardware specialized for PC were released in early 2012.[3] The official driver is available on the Microsoft website and it only requires one USB port and a PC with Windows 7 or above. Kinect provided developers with 640×480 RGB 30 FPS video stream and 320×240 30FPS depth map stream. We want to push the limit of Kinect with Computer Vision and Computer Animation algorithms and get as accurate movement as possible to simulate the sensitiveness of the real ultrasound devices. And combine into a ideal low-cost, realistic training device.

[1] : http://www.bluephantom.com/

[2] : http://www.vrealities.com/pcibird.html

[3] : http://www.microsoft.com/en-us/kinectforwindows/ 

For complete design document go to:
https://docs.google.com/open?id=0B21htwSnbMf0LTZybUFaUXo4MTA