You should convert the files to mp4 using a free utility like Handbrake.

There is no particular file size limit in Pinnacle Studio, but it does not seem to do well with video game recordings.