Learning Sign From Subtitles: A Weakly Supervised Approach to Sign Language Recognition.

Due to the lack of accurately labeled data available sign language recognition is often only based on a small vocabulary of signs with a small subset of signers and dialects. In response to this we have started to look at using the data available from the BBC, this data is broadcast with an inset signer and subtitles of the spoken content. We propose that it is possible to find correlations between what the signer is signing and what the subtitles are saying to learn signs and augment current data sets.

A brief look at this system is given over the following pages:

Features & Quantisation - How do we describe a sign.

Data Mining - The learning mechanism used to find correlations.

Localisation - How to use the correlations to find a sign.

Contextual Negatives - Why it's important what negative data you use.

Results - Word Spotting - Showing we can find a sign we know is there.

Results - Weakly Supervised - Showing we can say where there's a sign even when we're unsure of its occurrence.

.Figure 1 - Overview of the learning process from finding the subtitles and features, through the iterative mining-localisation process to the final sign locations.