Fred Truman
Design & Code Creative
Scope Proposal: Words in Your Mouth
April 12, 2011
Words in Your Mouth
The gist of the challenge is this: I want to develop a procedure and program that will allow me to take a fairly standardized interview video clip and corresponding transcript as input, and create a Max-Headroom-esque output of the interviewee saying anything we choose within the obvious limits of the vocabulary set available in the interview. And put it on the web.
Current Barriers & Constraints
- Video composition. For this effect to work properly, I’ve been searching for interview clips where there is a single speaker positioned in a generally static way and shot from a single camera angle throughout the entire clip. The reason being is that the abrupt changes in intonation when words are jumbled out of their original sentence contexts will prove jarring enough towards reasonable comprehension without the added parameter of a person jumping all over the screen and shown from different angles.
- Typical subtitle granularity. If you’re lucky enough to find a video that fits the above criteria, and it has been accurately subtitled, the level specificity is never more accurate than the sentence. This presents the largest and most obvious challenge and that is how do you generate a transcription of a video annotated with time codes at the word level. Given this data it would be easy enough to take that data and map it to the video clip.
- Video playback capabilities given a single clip (especially on the web). Even if you had this word-level specific transcription annotations, you would then face the challenge of how to play back each portion of the clip at the given time code. Currently the best options available for controlling a single video timeline on the web is popcorn.js, and its currentTime function may not perform at a high enough level to make this task possible. The other option, which would add and extra step between this and the previous, would be to take the transcription data accurate to the word level and generate some process to chop a video clip into a number of individual clips, each containing a single word utterance. Then, on the web, you might preload all of the clips within the given phrase before playing them back, and you could easily just swap their depths when transitioning between each.
The obvious alternative to this goal is to work with what is available, and that is subtitles at the sentence level. I might then be able to n-gram analysis at the sentence level and generate interesting conversation mashups. Even this goal has it’s challenge, which is how to take the start time for a subtitle and accurately estimate the duration of the entire utterance based on the character length of the subtitle. The good news is that this approach would definitely be more forgiving to slight inaccuracies in contrast to the word-level approach.
Before I get to the alternative, I’m trying to milk the original task as far as it will go before lowering the bar slightly. Right now, I’m looking into the possibilities of using Sphinx4‘s aligner audio file transcription process to generate time codes for each word in an audio file. I can just take the audio track from the video I want to transcribe, it will be the same length and should work. I can imagine that this won’t be any better than getting a Google voice transcription of a voicemail. We know how that goes.
Then taking this data, I would ideally be able to get some kind of video chopper process to output a bunch of independent clips. I have no idea what’s available for this process as of now. Please let me know if you have any ideas. The final step if all this works is to make some kind of web app to call a given number of clips and play them in sequence and hopefully the effect is at least funny. I’m thinking of approaching this by using a tornado python script, but I haven’t prototyped this yet so I’m still open to other options for this portion as well.