Stage I - Viseme Detection using Moments

Using the same skin segmentation as in the grid features moments can be calculated on a frame by frame basis. An example of the feature vector along with the types of moments used is shown in figure 4. These features are used on an individual frame basis to classify hand arrangement (HA visemes) and on a multiple frame basis to classify hand motion (SIG visemes). An example of the 2D feature vector used for motion classifiers is shown in Figure 5a.


Figure 4 Example of a moments feature vector obtained for a frame.

Boosting then chooses from two different types of weak classifiers which act upon the 2D feature array; local binary patterns (LBPs) and additive classifiers. LBPs work on the gradient of a feature over time, they vary in size from 2 bits to 5 bits. The LBPs run parallel with the time axis so that they are always operating on one type of value. In essence, the LBPs encode whether a moment is increasing or decreasing with time. Examples of two 2-bit LBPs are shown in figure 5b. Figure 6 shows a more detailed example of how they are used on the 2D features with one example matching the LBP and the other failing on two gradients.


Figure 5 a) Example of multiple feature vectors over several frames and b) how the local binary patterns are applied.


Figure 6 Example of features matching and not matching an LBP

The additive classifiers sum the values across a single moment type for a given number of frames, they can be as small as a single value or as large as the maximum classifier size allowed. They therefore contain information about the magnitude of values across a given moment typewhich complements the LBPs gradient information

Some examples of the classifiers learnt are shown in figure 7. Note in a) the predominance of LBP increases as the eccentricity of the moments increases and in b) the predominance of LBP decreases as the eccentricity deacreases.


a) Hands Move Apart


b) Hands Move Together

Figure 7 Examples of two of the motion classifiers learnt for a) hands move apart and b) hands move together.