I’ll explain a typical use-case for both software to elaborate on your question:
You typically use Fraps to record the gameplay. As this is done *while* playing it does not typically use strong compression algorithms. This results in bigger file-sizes, but saves computing power.
After playing you use virtual dub to cut and re-encode the video you recorded with fraps.
You then use better compressing codecs for video and audio. The resulting file-sizes will be way smaller (magnitudes).
As an update to my original answer:
Dxtory is a good alternative to Fraps. Provides a lot more settings.
