Seek-previews for HTML5 <video>
4 September, 2012 § 7 Comments
Some popular video sites now show previews of the video while hovering over the scrubber. This is really useful for trying to find a position within a video without having to interrupt the current playback.
The other night I was wondering if this could be possible with HTML5’s <video> element. It turns out that something like this can be achievable, and with a couple bug fixes it can also be pretty performant.
The current method for implementing something like this for proprietary websites involves offline creation of a large sprite that includes all of the keyframes for the video. When the user is mousing over the scrubber, the web page can change the visible portion of the sprite to show the currently hovered frame. This works very well and is also very fast as all of the expensive computation is done ahead of time.
This offline method isn’t possible though for the open web today. The HTML5 standard doesn’t include a way for web authors to declare a keyframe sprite, and preexisting videos would lack support for it if this feature was added to the standard today.
Creating an implementation of seek-previews for HTML5 <video> isn’t that hard. To do it, I cloned the <video> element via element.cloneNode(true). I then set the cloned video element’s currentTime attribute to the hovered position within the scrubber. This turns out to work quite well, with the only drawback being poor seeking performance.
Setting the currentTime attribute involves trying to get the video playback position as close as possible to the requested time. If this position is far from a keyframe, heavy decoding will be required. Implementing a “fast seek” method would greatly improve the seeking performance here. Bug 778077 was filed to implement this “fast seek” method (this will not replace currentTime). The other performance issue that will be encountered is unbuffered frames. The method of seeking the video requires downloading the part of the video being seeked to, which is obviously slow.
Try out the demo and let me know what you think. I filed bug 787881 to implement this feature for Firefox. I hope that it will kick-start our fast-seek implementation, as fast-seek can also be used for our present-day seeking within the timeline.
JAWS 
That responds very slow where I am; the seeks complete before the preview appears in many cases…
Yeah, there’s still some performance things that we can do on the video backend for faster seeking, as well as some things that we can do in the front-end such as not trying to seek as often while mousing over.
Works quite well, though I have a few suggestions.
Block rendering thumbnails for sections without data, as leaving the cursor there will continuously update until complete.
Cache nearby frames so moving slowly on the seekbar doesnt take ages to generate new thumbs.
FWIW Gecko already uses fast (to the nearest key frame) seeking on WebM, so implementing bug 778077 would only help with audio formats and platform dependent decoding (via gstreamer).
Is this actually necessary? What’s the use case for wanting to jump to another part of the video without interrupting playback of the current video? When I first noticed the difference in seek behaviour between Flash and HTML5 videos on Youtube, I assumed it was designed to cover up a functional deficiency in Flash. Scrubbing through the actual full size video, backwards and forwards, works much better for me in HTML5, basically instantaneous. I just went and tried it and in the Flash version Youtube won’t even let me scrub anymore. It forces me to use the seek preview bar, which looks cool but is useless if I’m looking for anything small. If I try on Vimeo via Flash though I get the stuttery, laggy scrubbing behaviour I remember.
Regarding your implementation, if I start at one end of the bar and move quickly along, the thumbnail shown remains far longer than it should if it’s not ready to replace it. So while useful for pointing at a section and saying “show me what is here at this point” it’s not very good for “I know what I’m looking for, I’ll just scrub along till I see it” as you can easily pass what you were looking for without it being shown.
Very cool!
You know the precomputed method is prefectly possible with HTML5, you just have to write your own controls? Youtube has had this for a little while now.
Still, it would be a very nice addition to the built-in controls if we can make it performant.
My idea is that we download the video in small chunks at the same time similar to the P2P client i.e bittorrent. then we use one frame shot from each chunk to represent sprites respectively.