What has higher priority for making a PC that can record games nicely?

The gt 730 supports shadowplay do you use that to record?
The fact that you get good gameplay but choppy recording is due to task priorities in windows (google to see how to change them) you can put your recording software to real time to get smooth recordings but your FPS in-game might suffer.
Shadowplay has very low CPU overhead and should not affect your framerate.

so...
1) I don't think your computer is strong enough to record gameplay well at all, it's too low in all areas.
2) Rather than upgrade your old outdated hardware with somewhat stronger outdated hardware, I'd say start saving up around $800-900 and you can afford a system that can actually record stuff.
3) Your hard drives may not be good enough to record gameplay, especially if you try to run your games off the HDD you're recording too, they're also pretty dumpy HDDS.
4) You can try the following to see if it improves anything, try both the local recording setup method and the nvidia encoder setup method. The local method uses CPU power, which you don't have a lot of, and the nvidia encoder uses GPU power (but may require shadowplay functionality which you lack)

You can try OBS since it's free:
https://obsproject.com/
And here's how to set it up for local recordings, start at the recommended crf 15, and lower the number for higher quality and higher file size recordings, or raise it for slightly lower quality and smaller file sizes.
https://obsproject.com/forum/resources/how-to-make-high-quality-local-recordings.16/
And to set it up for twitch:
http://help.twitch.tv/customer/portal/articles/1262922-open-broadcaster-software

If you have an Nvidia GPU, you can use that to reduce the load on your CPU:
To set it up for recordings:
1. go to settings
2. go to encoding
3. click the Nvidia NVENC button
4. Make sure Use CBR is turned on
5. Make sure Enable CBR Padding is turned off
6. Set your Max Bitrate (Kb/s) to your upload max upload speed. You'll want like 8000 or more though, this affects the visual quality of the video partly. (1mbps = 1000 kbps, www.speedtest.net)
7. go to Video and set the FPS to 60
8. Go to Advanced
9. Turn Use Multithreaded Optimizations on
10. Set NVENC Preset to High Quality
11. Set Encoding Profile to Main
12. Turn Use CFR on.

To capture your game:
1. Right click in the white area of the Scenes: box and choose add scene, name it after the game you're gonna play.
2. Start the game you're gonna play.
3. right click in the Sources: box and choose Game Capture (for full screen games) or Window Capture (for windowed games whether borderless or not) or Monitor (to capture whatever is display one your monitor including your desktop/webbrowser/OBS)
4. Start recording.

To add a webcam:
1. Right click in the Sources: box and choose Video Capture Device.
2. Choose your webcam from the dropdown list at the top. Press Okay.
3. Click Preview Stream
4. Click the Video Capture in the sources list, go to Order and choose Move to Top.
5. Press the Edit Scene button

you'll definitely need a bigger PSU as well, but the size of yours makes me think it may be an external one. (does it have a laptop like power brick? or is it just a straight cord from the outlet to the computer?) you can't upgrade external PSUs.