Frankenstein HTPC - Bottleneck Analysis & GC Rec's

GPU going bad ??

Try kodi mate , its a small install , it has its own independent video codecs & playback engine with hw acceleration.
If it struggles then your problem is hardware based as kodi will manage 20gb+ bluray rips on even a cheap amlogic android box with Mali GPU.

Your problem seems to be graphics related, not cpu related.

Your 5450 is getting old.
One possibility is that it is starting to fail.
swapping it out for a GTX750 seems to me to be a good idea.
Do not pay much attention to the internal specs like gddr2/3/5 or even the amount of vram.
The specs will be appropriate to each card.

When it comes time to change the cpu, consider that your E2-3200 has a passmark rating of 1486 with a single thread rating of 824.
By today's standards, that is poor.
The lowest priced Intel skylake G4400($65) has a passmark rating of 3821, and a single thread rating of 1890.
And, the integrated graphics is stronger than your current discrete card.

Take out the graphics card for now, and just use the APU graphics.

If that works well enough for your video situation there's no need to upgrade anything.

Even if it works (no pixelation) but with stutter you've at least shown your HD5450 was the problem so in that case a different GPU makes sense, but get whatever's cheapest and does the job.

I have to go, but maybe check if H.265 is supported on the GTX750 or similar (but then an HD5450/6450 might cost $25 or so..)

^ I'm with photonboy on that idea - the igp on that apu is very similar performance wise to the 5450 when it comes to video playback.

Also try kodi mate , seriously its video playback engine is far far superior to vlc by a long stretch.

More info..
I don't use VLC as it seems to periodically show poorly decoded video (on Windows but same on Linux).

I do know there are settings to enable hardware acceleration (GPU must have decoder to support it like H.264). That reduces the load on the CPU which for you is likely the cause of STUTTERING.

I guess you use Linux, so open a tool to monitor CPU usage. It should be easy to Google that depending on your distro. So open that, make sure both GRAPHS are showing (dual-core CPU) and monitor that whilst playing a video. If it's stuttering you should see CPU usage hit 100% on at least one of the cores, maybe both.

If hardware acceleration is OFF then turn it on (media player).

VLC confused me a bit, but there's TWO sections of relevance under "Tools->Preferences", Video and Input/Codec (on Windows now). I'd enable anything that looks like "acceleration" as well as disable the "postprocessing" and see how that affects CPU usage.

*The above also applies to buying a new GPU. If you aren't enabling hardware acceleration (MPEG2, H.264, VC1..) then it's all processed on the CPU.

Other:
Some devices can be set to DECODE the video and just STREAM the video to your HTPC. Basically your HTPC just reroutes it to the HDTV.

I have a WD MY CLOUD which is awesome as it just plugs into the router thus can be accessed by all devices wired, or wireless on or local network.

I haven't tested the streaming capability though.

*I had a WDTV Hub (so do my parents and sister) and it was perfect for playing video. There's another version (Elements?) with no hard drive but it won't play Netflix. It does however play every video/audio codec I have which is great (most other media players don't).

If a simple WDTV media player is all you need that might be a better choice since it's simple, has a remote and just works. I've got a WDTV Hub in the basement, a Samsung BluRay player in my office and a WDMY Cloud to distribute video (4TB plus a backup 4TB WD Elements).

For the WDTV Hub I just turn it on and navigate to the video I want and play it. Eezy peezy, though the pictures don't work (metadata won't download to remote unit like on a local USB or internal hard drive).

My Samsung player won't play everything, like most media players. In fact, some of my BD rips start but the audio drops out due to Cinavia audio watermark protection.

I don't want to use my PC, so I think I'll either get another WDTV media player, or investigate an inexpensive Intel x86 SoC solution and use KODI-Linux. I'd prefer it to handle Netflix but I have that on my Samsung player so it's not a deal breaker. I think H.265 and HDMI2 support to future proof would be good.

Summary:
I'm just throwing out some ideas. Basically it's hard to find an all-in-one solution that's ideal. Build a PC and the interface might not work (though Kodi-Linux or similar might be fine with a compatible remote).

Buying a video card might be needed for your PC, but then if that costs about the same as a MEDIA PLAYER that can serve you as well or better why bother?

Anyway, start with my advice about the CPU usage, hardware acceleration on/off and take it from there.

^ he's using win 7 mate .
Hw decoding is going to be non-existant if those rips are encoded as h265/hevc - & with high bitrate bluray rips they likely are.
H265 hw decoding in not supported at all on current gpu's within windows or Linux for some reason.
Deciding is going to be entirely software based & while its CPU based decoding is fairly efficient vlc is still massively inferior to kodi's engine IMO.

Enen a ivy bridge i5 will manage hevc high bitrate encodes mate , or the 860k or any fx chip
You can go as cheap or as expensive as you like.
An haswell i3 will absolutely do the job fine though & price to performance is likely your best bet.