All Spacial Information Collected for Both Probe and Needle

The above video shows the effect that all of the markers of the probe and the needle are being tracked by the camera.

The user should wear white coat and also white gloves if necessary, because skin color and the needle marker colors are similar thus confusing the camera. We will try to modify the color alignment to make it more convenient, but this might also be reasonable for trainees. The background is also better to be white.

As we noticed before, the data collected vibrates a lot. We are not sure how this would be like in the mock ultrasound output since we haven't integrated our codes. However, we are also trying to use some filter( Karman Filter or simpler, low pass filters) for optimization.

Cheers!

Multiple Updates - Needle Rendering/Integration

Needle Visualization

Now we have the needle visualization along with the medical data rendering. In the video I am moving the slicing plane front and back, so the needle is visible sometimes (when the slicing plane and the needle are nearly in the same plane). As the slicing plane gets near the needle plane the needle visualization becomes brighter and brighter. Also note that the slicing algorithm is capable of changing the needle visualization as the needle penetrates into or off the body.

Integration Methodology

Robin and I have very specific settings for each of our modules, so we decided not to integrate the modules at the code level. So I thought it would be good if we make Robin's module send the data that it captures using Kinect to my module using Shared Memory. After a little searching in the internet, I got a surprisingly easy method to implement shared memory.

In the video above, I created a dummy process, "Data Provider" which just randomly sends data regarding the probe position and orientation. The rendering module is rendering as per the data that it gets from the dummy process.

Now we are going to add the small piece of code in Robin's module so that it can send data to my module.

Probe 3D Spacial information

Got 3D Spacial information for all of the markers and the bottom surface of the probe.

The values vibrated heavily because all of the inputs are unstable. We are now trying to integrate our code to see the final results.

In the video the bottom line shows the probe surface position.

Getting Probe Information

Now we are able to identify all the markers and the mock dummy. With the markers' 3D spatial information we are trying to get the position of the probe first. We measure the distance between the center of the markers and get:


blue-green: 116mm
blue-yellow: 83mm

We define a black circle which is the upperleft corner of the mock dummy and will be the relative zero point in the xyz cordinates. x axis points towards right, y points up and z points out the image.
We also require the three markers of the probe be positioned at the rectangles drawn in the image to identify. The board which carries the blue and green markers should be horizontal.

From the given data we want to find the relation between the pixel index and the 3D coordinates information, which includes 1)Up vector; 2) View direction for the ultrasound image

Probe Simulator

After we confirmed that we could get accurate depth data from Kinect, we attached three markers to the mock probe and tracked the 3D spacial information of them. Based on this information, we will be able to detect the position, direction and rotation of the probe.

Since the markers are smaller than the cups that we used previously, to make sure they are sensitive enough, their contrast against the background should be as high as possible. So we used Tianyu's black coat and it works very well. Also the board which carries the markers is dark yellow, similar to skin, so we moved the stand instead of the probe itself which would confuse the camera between our skin and the board. However, the light yellow marker could be detected even if we hold the probe in hand. So it could be the same convenient for the trainee.

We will modify the color of the board, record the requirement of the background and create a mock dummy with a box to test the 3D spacial data for the probe. After that, we will also try to integrate our codes.

Depth Data Detected

Using the method in the last blog, we tested the depth value of the objects. We displayed the values on the output images and as seen in the above video, they are pretty accurate and quick.

There are some occasions that the values are zero or 17** which is the depth of the background wall. The first error is because the shadow of the cup moves its mass center from its real center to the bottom, which is very like to be "invalid area"; the second error happens when the object is moved very fast, because depth image falls behind the color image by one to two frames and actually returns the depth value of the same coordinates in the previous frame.

To solve the first problem, we just need to cover the table with a low reflective cloth to remove the shadows; the second one could rarely happen in practice, but I will also filter it out.

Calibrate Depth Map and Color Map

There is some offset between the color image and depth map. In order to get the depth data, we got some sample images and calibrated them.

Color Image

Depth Image

Calibration

So In order to get the corresponding pixel in the depth image from the color image, we should firstly clip the color image out of the rectangle (52, 30) (626,460), then amplify the coordinates (x, y) by (1/0.9) to get the coordinates(x', y') in the depth image.

Track Multiple Objects

We are now using the CamShift algorithm from OpenCV library to track objects. This algorithm identifies one object from the color of the selected area and track it using the histogram. We upgraded it to track multiple objects in the scene. It works well and could also solve hidden situations. But since it is sensitive to color, we have to distinguish the objects from background and themselves. Now we are going to:

1) test the algorithm with up to five objects

2)design the markers and calibration process.

3)gain information from depth map

Below is the demo for two objects tracking

Results of Slicing Volumetric Data Constructed from Real Ultrasound Images

Slices of Volumetric Data constructed from Real Ultrasound Image

Finally, we can see some images which are very close to the target of this project. I am moving the slice plane using keyboard right now. But soon enough Robin and I will start working on integrating this Rendering module with the Kinect Tracking module.

Optimized Tracking Algorithm

The above video shows the effect of the new tracking approach. This method is tracking the mass center of each object frame by frame. It is more accurate and faster, and it could return the mass center of each object, which is helpful to get the depth data from kinect's depth map. However, it cannot solve the hidden problem. And currently it cannot be run in release mode. Trying to figure it out. Also discussing with others about the possibility of using existing algorithms like Camshift.

Slicing Test Cases and Rendering in OpenGL

Slices of Cylinder Along Y(-Z) axis

Rendering Slices of Cylinder Along Z axis in OpenGL in Real Time

In the last update I had the slices of spheres and cloud. This time to really test the slicing algorithm, I used a cylinder and sliced it along many different directions. Along Z axis, there are rectangular slices. Along Y axis, there are circular slices. Along Y(-Z) there are elliptical slices. Thus, the slicer passed all the tests.

Further, I have used OpenGL for rendering the results dynamically. First I tested with a 100 x 150 image. It was running real time at that resolution. But then I increased the resolution to 510 x 335, at which the slicer became really "not real time".

Thankfully, I had spent time in the earlier stages of the project working on multi threading. I added multi threading to my slicing algorithm and now it runs in real time!

Multiple Object Tracking

We wrote a method to track multiple objects in the view scene. In the movie below, two different objects are identified as blue and black markers( from their original positions). Then they are tracked as being moved. However, the tracker sometimes has errors, for example, they can sometimes misidentify the blue one with the black one. The frame rate is also kinda low. Improving them. Trying to optimize the algorithm by estimating and tracking the mass center of each object.

Tracking with Original Calibration

We would like to track the different markers via identifying them at the original place. Thus we fixed the camera and the different objects as the image shows below. The original position of the markers are shown in the 2D camera image as:
GREEN:(187,266),(205,315)
RED:(463,292),(505,310)

The two coordinates are for the upper left corner and lower right corner of the two rectangles. We will use a matrix called identifier to define and track the objects in the view scene.

Sensitive to light

This morning the code worked pretty well, then in the afternoon it didn't. After a while I found out that the difference is that the light was turned on(see the circled area in the image below). So the corner detector is pretty sensitive to light in the scene, especially the array of lights which also form some kind of corner pattern. Take this note for further calibrations.

Marker Detector

Modified the marker to the Chessboard pattern.

Append the Marker to a pen so the cylindrical pattern could be seen from every direction.

The result is decent, as seen from the two images below.

Within the distance that is reasonable for the accuracy of Kinect Depth Sensor, both of the pens could be detected from the scene. More patterns would be added like color to identify between two objects.

More details will be shown later today or this weekend.

Kinect Depth Map Calibrated

Calibrated the Kinect Depth Sensor.

Picked the depth value of the mid point of depth map. Measured with A4 paper whose length is 279mm.
Data collected as below:

1674:A4*6-1661
1395:A4*5-1382
1116:A4*4-1111
837:A4*3-836
558-A4*2-NULL


In each row the first number is the actual length of the string of paper; 6/5/4/3 is the number of paper; the last number after "-" is the value is abstracted from Kinect Depth Sensor's return value. The value returned should right switch 3 digits(these 3 digits contain the index information).


So the sensor works pretty well in detecting the distance from the object. Especially between 800 to 1100, the error is limited within 5mm. That is the best area for Kinect to detect.