This is my favourite way to retime footage from 30 fps to 25 fps in high quality with After Effects. This method does not use any 3rd party plugins like Twixtor. Off we go:
Import your final film. It should have been edited at 30 (or 29.97 if you converted to cineform and maybe in some other cases too) fps. Go to the footage in the project window and choose Interpret footage. Set the frame rate to your target frame rate (in my case 25 fps).
Create a new comp based on this footage. It should be set to 25 fps as well. If you take a look then you will see that what is happening now is that we play the footage back with a lower speed. To correct this, apply the TimeWarp effect. You could use the speed parameter, but I prefer to use to keyframe the source frame parameter. Change your project settings to display frames instead of a timecode:
Note the length of your footage in frames. Now calculate the new end point: <number of source frames> / <original framerate> * <target framerate>. Round up if this results in a fraction instead of an integer number. Go to the frame that you calculated (which should be exactly at 5/6th of the total length of your footage/timeline). Add a keyframe for the Source Frame parameter here and set it to the last frame (the number of source frames). Go to the beginning of your comp and keyframe to source frame 0. Set the method that Timewarp uses to Frame Mix (for frame blending) or Pixel Motion (for frame generation). If you scroll through it now (can be very slow with pixel motion) you will see the retimed footage.
If you don’t have any hard cuts, you’re done now and can render the result. If you have hard cuts, you will want to protect them.
You can go to all hard cuts, keyframe it to one exact frame just before the cut and keyframe it to exactly one frame later after the cut. But this is tedious, so here is a quicker way:
Find all hard cuts and place layer markers there (or let the very nice Magnum script do the work for you). Then add this expression to the source frame parameter:
n = 0;
t = 0;
if (marker.numKeys > 0)
{
n = marker.nearestKey(time).index;
}
if (n > 0)
{
t = (time +thisComp.frameDuration) - marker.key(n).time;
if (Math.abs(t) <= thisComp.frameDuration)
Math.round(effect("Timewarp")(4))
else
effect("Timewarp")(4)
}
else
effect("Timewarp")(4)
It will keep the animated value unless there is a layer marker present at the current or the next frame in which case it will round the time to make sure that no blending across the hard cut occurs.
In a future post I will show a way to do the same thing in twixtor which handles the hard cuts differently.
If you get messed up frames in between you can either switch back to frame blending for the few frames that require it or use brainstorm to optimize the optical flow analysis parameters of Timewarp.
As the 5D mark II can still only record in 30 fps because there is no firmware update to record in more useful 25 fps (for pal DVD/TV) or 24 fps (for cinema projection), the need to retime footage from 30 fps to 25 fps definitely still exists. There are many methods to do this, but many are not as high quality as one would like. In this blog post I will explain what methods exists for retiming exist, what they look like and in the next follow up post I will show how to do high quality retiming in After Effects.
First, an overview of the problem: Your camera records in 30 frames per second (fps). You get the stuff into your editing software of choice, set the timeline to 30 fps and edit you film. You export your final film and up to here, everything is fine. But now you need a Pal DVD (to be played in Europe) or a film copy to screen in cinemas (lucky bastard). So now you have the problem of how to go from 30 fps to 25 or 24 fps.
If you had shot a silent film, you could simply play it back a little slower. A hypothetical 5 second film at 30 fps could be played at 25 fps and now be 6 seconds long. Everything would move a little slower, but maybe that could be cool. Unfortunately, if you try this with a film that uses audio, the sound would have to be slowed down as well and thus be significantly lower in pitch.
Since most films nowadays want to use sound, this leaves us with 3 options that slow down the images while maintaining the same duration and thus keep the audio unchanged.
All following samples will use PALs 25 fps, the same methods can of course be used to go to 24 fps, just the numbers are slightly different. The sample videos linked to below are all the same clip of some location scouting footage that I edited specifically so that one could study the effects of different retiming methods. Here is the original clip at 30 fps (29.97 to be exact because I converted it to cineform via NeoScene)
Frame dropping
The first option is frame dropping. It simply removes every 6th frame. Our 5 second clip from before would still be 5 seconds long, but of the original 150 frames, 25 would have been dropped. Here is a little graphic I made to illustrate what happens.In this example we see 6 frames at 30 fps that need to become 5 frames at 25 fps (so that at both frame rates, the clip would play in exactly 1/5th of a second or 0.2 seconds).
Frame dropping leads to a strange jerky sensation that can be seen here. Some people are more sensitive to this than others:
Advantages of this method
It works consistently
It renders very quickly
No problems with hard cuts
Disadvantages of this method
Looks very strange
Frame blending
Frame blending always blends two frames together. So the first frame A of our new 25 fps sequence is a blending of frame A and B in the original sequence; the next frame is constructed from B and C and so forth (the exact details how and which frames are blended can vary). Here is an illustration of what happens:
In this graphics this looks pretty bizarre, but since most individual frames of a film look very much alike, this usually works pretty well. It gets more noticeable when there are quick movements, especially when the scene was filmed with a short shutter time and the objects have little motion blur. Here is a frame where the blending can be seen very well:
The good thing about frame blending is that it looks relatively pleasant most of the time and it works in any case. One of the problems with it is that it will soften your hard cuts. Here is an example where what used to be a hard cut from this frame
To this one
Gets turned into this blended frame
Here is the sample clip with frame blending.
In after effects one can protect hard cuts (see this post for more information on how to do this). The resulting sample can be seen here:
Advantages of frame blending
It is more pleasant to watch than frame dropping
It renders quickly
Most of the time it looks ok
It works consistently
Disadvantages
Extra care needs to be taken to protect hard cuts
Noticeable "double exposure" effect
Frame generation
Frame generation is a process that goes by a myriad of names (aka Optical Flow Analysis, Pixel Motion, Motion Vectors, ...). The principle is always the same though: a clever algorithm tries to understand where similar groups of pixels move in the image and calculates in between frames by moving the pixel groups further than they used to move, an extra 1/6th of the original distance to be precise (in this particular case). The upside is that if it works, the image looks perfect and no matter how much movement is in a frame, there will never be the "double exposure" effect seen with frame blending. The downside is that even though these algorithms have gotten very good, sometimes they get badly messed up and produce something like this:
This happens basically at every hard cut, so you have to protect your hard cuts against this problem (see this post for more information on how to do this). However, it can even happen when there are no cuts, e.g. when the lighting changes dramatically (see this blog post for an image where that happened). If that happens, tweaking the parameters can often help, but sometimes you will have to resort to frame blending for a couple of frames where frame generation simply does not work.
Here is an example with pixel motion where hard cuts are unprotected (this sample uses twixtor instead of After Effects, but the results in this case are very much alike):
And here with hard cuts are protected (native After Effects again):
See the bottom of this post for some more clips retimed with the Twixtor plugin.
Advantages
If it works, the quality is very good
Disadvantages
Extra care needs to be taken to protect hard cuts
Can sometimes produce messed up frames, output has to be carefully watched and problems eliminated
Longer rendertimes
My advice would be to go with frame blending if are in a hurry, and use frame generation if you have time to check and maybe fix the output and want very high quality. In both cases you should remember to protect your hard cuts – something that to my knowledge can only be done in compositing programs like After Effects. If you have the option to use Twixtor, I can highly recommend it as it seems both more stable and significantly faster than TimeWarp. Details on how to use After Effects for retiming can be found in my next blog post.
For the sake of completeness, here are 3 samples of the result when using Twixtor with frame blending and frame generation:
I just made my sister very happy by recovering lost photos from her last holliday that were still on a flash card but unreadable. If you ever need a tool like this the best one for this job in my experience is PhotoRec. It doesn't have a great user interface, but it works very well and is available on a number of plattforms. If you don't know exactly what option to choose when it asks you something, just pick the default (just be sure to select the right drive to recover from and a folder on your hd to save the recovered images to).
As I wrote in my previous blog posts (post 1, post 2) I am currently testing several methods to see if video material from the 5D Mark II can be converted to 25p. I also uploaded a couple of shots to youtube, links are below.
Being unable to shoot 25 or 24 fps video is, in my opinion, still the biggest shortcoming of the 5D for filmmaking. From my experiments, I would say that retiming using optical flow analysis is a workaround that will work automatically for probably 80-95 % of your short film. It will exhibit some artefacts in the rest. Depending on how sensitive you are about motion artefacts you may or may not notice them when viewing the film the first time. By adding time in the post production workflow to work around the problematic areas by either helping the optical flow analysis with manual foreground extraction or by using frame blending in these cases one should be able to arrive at a decent 25p copy that will feel right to a very large part of your audience.
Stu Maschwitz nailed it when he said: "That's not a workflow. It's finger-crossing" (link to the forum posting). But given enough time and dedication, it IS possible to make almost any material work well enough, so it is a viable alternative to shoot with the 5D and hazard the consequences (long post production, some shots may always feel weird). But, in a way, No Budget filmmaking has always had to make tradeoffs like this.
Now for some of the tests I shot with DoP Philip Leutert, all retimed with Twixtor in After Effects unless otherwise noted:
A lot of the shots that I thought would be tricky worked surprisingly well. Here is an example of a pond and a duck. Here is another one where you can see some detailed smoke.
The most problematic type of shots for optical flow analysis is when a foreground object occludes and reveals a background that is clearly visible and has very noticeable detail - human faces are the worst background because we know so well how faces should look like. Here is an example where I walk around a bit. Watch the outlines of the foreground and how it distorts the background. Here is another example where my face is revealed and occluded a couple of times.
Semitransparent objects in front of a face are also tricky, but in my test twixtor held up rather well. (And yes, I do look silly holding up some semitransparent stuff in front of my face). Here is the same material retimed using frame blending.
The results for me personally are encouraging enough that I would risk shooting a short film with it. It will take a lot of time to get a good 25p copy, but at least I think it is doable. I think the resulting material will never be perfect, but if the story, the sound & the music are good then people shouldn't be thrown out of their experience - and the shallow dof of the 5D can be a very important story telling device, so it may just be the right choice for some projects. Of course, we should all be writing to Canon to stop being stupid and give us 25p (24p would also be nice, but it's not as crucial).
Let me know if you would like to see some more material or if you are interested in alternative techniques (After Effects' Pixel Motion or frame blending) or if you have a copy of Shake and would like to try to retime my test material with it!
I will shoot some more tests with the 5D tomorrow with DoP Philip Leutert and post more results soon. (Update: blog posting about these tests is here)
Meanwhile, here is a little trick on how to find optimum settings for time warp in After Effects: use brainstorm. It shows you 8 new variations of the selected effect settings, which is ideal to quickly find those settings that remove obvious artifacts. Select the parameters "Vector Detail", "Global Smoothness", "Local Smoothness", "Smoothing iterations", "Error Threshold" and "Block Size", then press the Brainstorm button (at the top of the timeline):
A few fellow students sent me some comments and links which I want to repost here:
Some people think that Shake has the best optical flow retiming, others seem to think Kronos (from the Foundry) is best (which is odd because this is what After Effects uses in the time warp effect and that one seemed to have some pretty obvious problems with our test footage) some say Twixtor is best. I'd love to see a comparision of Shake and Twixtor, does anybody know one?
