Your cpu is a little on the older side, but it should still handle 720p. Since you have a decent graphics card, you should try Media Player Classic Home Cinema, which allows you to offload most (if not all) of the decoding process to the gpu: http://www.guru3d.com/article/accelerate-x264-1080p-mov...
That video card can help decode AVC/H.264, VC-1 and MPEG-2.
You must enable that capability in your mediaplayer:
1) Boxee (Windows) can check/encheck hardware decoding in the settings
2) VLC setting is under "preferences -> inputs and codecs -> "use GPU acceleration"
3) K-Lite package (Media Player Classic HC) "view->options->internal filters"
1) I recommend the K-lite package (free-codecs.com) STANDARD package. Install the 32-bit version. If you also want to play using the Windows Media Center and you also have 64-bit Windows you should ALSO install the K-Lite 64-bit package (so just the 32-bit for 32-bit Windows and both for 64-bit)
2) I have an HD5870 and GPU acceleration was working perfectly until Catalyst 10.9 or 10.10 and then the video looks bad especially in darker areas. I rolled back to 10.7 and it looked fine. This problem of washed out colours and grainy dark areas still exists in 10.12 so I'm NOT using GPU acceleration until they fix it.
3) You can monitor your CPU usage using the Task Manager (CTRL-ALT-DEL). Start a video and see if it spikes to 100% which is likely the cause if you have stuttering.