My research for the Masters Degree consisted of the extension of an algorithm for considering image segmentations in a probabilistic framework. The algorithm was developed by Steve LaValle for segmenting range images and was extended by Becky Castaño to considering grouping of symmetries in line drawings. I changed the method of integration from a numerical integration using Monte-Carlo methods to a summation over known samples, and implemented the algorithm in C++ to segment textured intensity images. My implementation uses a Markov Random Field model for texture. One thing this algorithm does is allows the segmentation algorithm to evaluate many possible segmentations of an image, and return the N most probable of these. A page from my thesis illustrates this concept. The abstract from my thesis follows.
Traditionally, the goal of image segmentation is to produce a single partition of an image. This partition is then compared to some ``ground truth'', or human approved partition, to evaluate the performance of the algorithm. This thesis utilizes a framework for considering a range of possible partitions of the image to compute a distribution of possible partitions of the image. This is an important distinction from the traditional model of segmentation, and has many implications in the integration of segmentation and recognition research. The probabilistic framework which enables us to return a confidence measure on each result also allows us to discard from consideration entire classes of results due to their low cumulative probability. Several experimental results are presented using Markov Random Fields as texture models to generate distributions of segments and segmentations on textured images. Both simple, homogeneous images and natural scenes are presented.
I believe that it is important for there to be more communication between vision modules than is possible in the typical segmentation goal, which is to return a single segmentation which is best in some sense. For example, if a satellite picture of crops is being looked at in the wrong spectral band, no information (or not much) will be available. The segmentation algorithm should note this and communicate this information to the higher level algorithm, which might try another spectral band or might run a test to see which band to look in. The idea is simply that often a single image isn't enough.
I've continued my work in uncertainty representation into the area of object tracking. My current work utilizes an optimal recursive data processing algorithm called the Extended Kalman Filter (EKF) to use information about the uncertainty in feature tracking and the uncertainty in the state estimate of an object to aid in tracking that object. For example, if I'm looking at the back of your head, I have much less confidence in my estimate of where you are looking than if I'm looking at your face. I have formalized this idea and implemented it into an object tracking system. Some example sequences are shown below.
The examples below look best on a 24-bit monitor. They seem (at best) a bit dim and grainy on an 8-bit (256 color) monitor. Sorry about that.
|
|
This sequence shows the effect of external occlusion on feature tracking. One feature of my system is that it is (somewhat) robust to unexpected occlusions. The input to a feature tracker is shown in the upper left of the sequence, and the feature tracking results are shown in the lower right. The results are in the form of an SSD (Sum of Squared Differences) dissimilarity measure for the image region around the gripper. In other words, it's a graph of how much the area around the gripper looks like our a-priori idea of what a gripper is supposed to look like. In the normal case, there is a single well-defined hump in the center. This is a case of good feature tracking. When the gripper becomes occluded, nothing in that area of the image matches our template well. Thus, there are no well-defined humps in the SSD surface. We scale this surface, and model the location of the tracked feature as a 2D Gaussian random vector. We use the location of the hump as an estimate for the mode of the random variable, and the shape and orientation of this hump to estimate the covariance matrix for the random variable. When the feature is occluded, hump flattens out, the covariances become large, and we can tell that the tracking is not doing nearly as well. With this information, our tracking algorithm can discount feature tracking results such as this, and successfully track under conditions of occlusion. |
 |
This sequence shows the feature points used to track a 3 degree of freedom PUMA robotic arm. The selection of these feature points is manual. They are chosen to attempt to ensure that the motion of each joint is observable in every configuration of the arm, through at least one feature. |
 |
This sequence shows the tracking of an arm with 3 degrees of freedom active. In this sequence, we illustrate tracking with a fixed, known, set of features. This is the usual scenario for trackers. Several points on the side of the arm are used to help estimate the joint angles of the arm. In this sequence, the image in the upper right is the input to the tracking algorithm, and the image in the lower left represents a display of the estimated joint angles. Remember, the only input to the algorithm (other than an object and kinematic model of the arm) is the upper right image). |
 |
This sequence shows the tracking of an arm with 3 degrees of freedom active. At the beginning of the clip, some features on the arm that are used for tracking (for example, the gripper fingers) are occluded by another part of the arm. Since we have an estimated configuration for the object, we can predict this using our model for the arm. When the models predict that the gripper will become visible, later in the sequence, those features begin tracking and help the estimation process. |
| This sequence shows the tracking of an arm with 3 degrees of freedom active, when some features are occluded. Note that the algorithm does not know that features are occluded, but has to use feature tracking results (as described above) to discount bad feature tracking results. When the occluded features are tracked, the estimated variance will be large, and the EKF algorithm takes this into account when assimilating the feature tracking results into the object configuration estimate. Note that if a feature is discriminating well in one direction, but not another (such as a edge feature), the algorithm will behave appropriately by discounting the effect of the feature tracking along the edge while using the tracking information across the edge. |
 |
This sequence shows the tracking of an arm with 3 degrees of freedom active. In this clip, we allow features to come into and out of the active feature set, significantly increasing the motion of the arm that we can track. As described above, the estimated configuration of the arm determines which features are expected to be visible at any point in time. Even if a feature that is expected to be visible is not, the algorithm does not fail drastically. In this case, the math works out the same as for the case of externally occluded features. Therefore, a feature that is not tracked well (because it is not visible) is discounted when assimilating tracking information into the configuration estimate. |
 |
This sequence shows the tracking of an arm with 2 degrees of freedom active. The lengths of each link of the arm are initialized correctly, so the tracking progresses much as if the link lengths had been measured and set. In the estimation process, which is based on the same mathematically framework (extended Kalman Filtering) as the previous cases, the joint angles are modeled as moving with constant velocity and the link lengths are modeled as being unknown constants. The sequence above illustrates the initiation of tracking, but the full sequence is also available. |
 |
This sequence shows the tracking of an arm with 2 degrees of freedom active. The lengths of each link of the arm are initialized incorrectly, so the tracking progresses much as if the link lengths had been measured and set. In the estimation process, which is based on the same mathematically framework (extended Kalman Filtering) as the previous cases, the joint angles are modeled as moving with constant velocity and the link lengths are modeled as being unknown constants. The sequence above illustrates the initiation of tracking, but the full sequence is also available. |