A Unified Framework for Concurrent Usage of Hand Gesture, Shape and Pose

Natural interaction with hands

• hand trajectory
• HMM based continuous recognition
• not really natural
• hand shape
• Use depth image directly / label / recognition
• articulated hand pose
• Hand skeleton extraction

[Download][Summary][Slides]

One-shot gesture recognition

Data
User dependent:  the gestures in one batch are performed by a single user. There is a single labeled training example of each gestures of the vocabulary in a given batch. The goal of the challenge is, for each batch, to train a system on the training examples, and to make predictions of the labels for the test examples. The test labels in the validation batches are withheld. Additional batches finalXX will be provided for final testing.

Overall analysis
https://docs.google.com/file/d/0B08QS7nJpK7mX3pFalVzckxoU2M/edit

Winners' methods
1st place
https://docs.google.com/file/d/0B4jW8HPqnNiuU2RiQWl6TnpfQzQ/edit

Initial image preprocessing

• used depth information only
• Identify outliers (those pixels returned as 0 by kinect) and remove outliers (simple/fast wavelet reconstruction)

Representation of visual features

• Mimic behavioral and neural mechanisms underlying visual processing
• Selected features of interest (emphasize moving close to the camera gestures)
• Feature/background separation
• Encode features time-varying shape and trajectory
• Similarity measure (robust to variability in features selection or location)

Gesture recognition algorithm

• General Bayesian network model similar to speech recognition literature
• Can perform simultaneous recognition and segmentation
• Compute similarities between each input video frame with sample gesture video frames

2nd place

Gesture classification pipeline

There are a couple gesture classification pipeline which provides a clean and nice framework. Here I am going to list their basic structures.

• Gesture recognition toolkit
• pipeline
• pre-processing
• feature extraction: should be chainable
• classifier
• post-processing 
• methods
• train
• test
• F measure
• precision
• recall

Mahalanobis distance and Gaussian distribution

The Mahalanobis distance is a distance measure that accounts for the covariance or "stretch" of the shape in which the data lies.

$$D_{\text{Mahalanobis}}(\mathbf{x}, \mathbf{y}) = \sqrt{(\mathbf{x} - \mathbf{y})^T\Sigma^{-1}(\mathbf{x} - \mathbf{y})}$$

This is very similar to the exponential term in the Gaussian distribution.

It is useful to get an intuitive feel about the standard deviation. In the one-dimensional case, where
$$ p(x) = \frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{(x - \mu)^2}{2\sigma}}$$
the probability at one standard deviation away from the center is \(\frac{1}{e}\) (37%) of the peak probability.

Source:
https://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg

Go and Scala

Start to learn Go and Scala. I'm going to jot down some notes here.

Both Scala and Go have similar type declaration syntax, with type name on the right.
Scala:

class Person(first: String, lastName: String)

Go:

func needInt(x int) int {return x * 10 + 1}

Machine learning

Perceptron

Repeat until convergence:

For t = 1 ... n

1. \(y'\) = sign(\(\underline{x}_t\cdot\underline{\theta}\))
2. If \(y'\ne y_t\) Then \(\underline{\theta} = \underline{\theta} + y_t\underline{x}_t\), Else leave \(\underline{\theta}\) unchanged.

Kernel form of the perceptron

• Definition: for any \(\underline{x}\), define \(g(x) = \sum_{j=1}^n\alpha_jy_jK(\underline{x}_j,\underline{x})\) where \(K(\underline{x}_j,\underline{x}) = \phi(\underline{x}_j)\cdot\phi(\underline{x})\)
• Repeat until convergence:
• For t = 1 ... n
1. y' = sign(g(\(\underline{x}_t\))
2. If \(y'\ne y_t\) Then \(\alpha_t = \alpha_t + 1\)

Algorithms for inference

Definitions

Chordal graph: a graph is chordal if any cycle of the graph of size >= 4 has a chord.

Formulas

Parameter estimation for HMM with Gaussian distribution for emission probabilities:

$$\underline{\mu}_s^t = \frac{\sum_{i=1}^n\sum_{j=1}^mp(S_j = s|\underline{x}_{i,1}\dots\underline{x}_{i,m};\theta)\underline{x}_{i,j}}{\sum_{i=1}^n\sum_{j=1}^mp(S_j = s|\underline{x}_{i,1}\dots\underline{x}_{i,m};\theta)}$$

General form of EM algorithm

The basic idea is that we want to maximize the likelihood of the "complete" data. Because we don't know the "complete" data, we need to take the expectation of the "complete" data with respect to the probability distribution of the hidden variables.

Let \(\mathbf{Z}\) denote all the hidden variables for all the examples, and let \(\boldsymbol{\theta}\) be all the parameters for the probability model. \begin{aligned}\boldsymbol{\theta}^{(i+1)} &= \underset{\boldsymbol{\theta}}{\arg\max}\sum_{\boldsymbol{z}}P(\boldsymbol{Z} = \boldsymbol{z}|\boldsymbol{x}, \boldsymbol{\theta}^{(i)})L(\boldsymbol{x}, \boldsymbol{Z} = \boldsymbol{z}|\boldsymbol{\theta}) \\ &=\underset{\boldsymbol{\theta}}{\arg\max}E_{P(\boldsymbol{Z} = \boldsymbol{z}|\boldsymbol{x}, \boldsymbol{\theta}^{(i)})}L(\boldsymbol{x}, \boldsymbol{Z}= \boldsymbol{z}|\boldsymbol{\theta}) \end{aligned}
The E-step is the computation of the summation, which is the expectation of the log likelihood of the "completed" data with respect to the distribution \(P(\boldsymbol{Z} = \boldsymbol{z}|\boldsymbol{x}, \boldsymbol{\theta}^{(i)})\).