Upgraded PC seems to be under-performing, various potential factors, please help!

This is a tricky one... A month back I created a thread inquiring about my computer, you guys on here answered well and I bought myself a new motherboard and CPU, My current spec is as follows:
Motherboard - Asrock Z87 Pro4
CPU - Intel i5 4670K
GPU - AMD Radeon 7950
RAM - 16GB DDR3
HDD - 1TB Seagate
OS - Windows 8

The scenario:
I bought this PC second hand last year, it seemed to run everything fine back then (I had a FX-8100 and some £30 motherboard), over about 6 months, the computer starting worsening, so I made a costly upgrade last month. It seems to work fine now, I can run pretty much anything I throw at it on the highest settings, at a constant 40-60 fps.

The problem:
Recently I've been coming across some issues, for example, earlier today I have tried installing this rather large modpack via dropbox onto minecraft, my friends are able to do it (some have lower performance rigs than mine, and one a laptop), yet mine cannot handle it. Although I have 16GB of ram, it says "run out of memory", I have done plenty of research on how to allocate more memory etc etc, but I have noticed something quite weird: On task manager, the CPU usage will fluctuate up to 100&, and the Memory usage will fluctuate up to around 50%, even though only about 1GB is being used. I defragmented my HDD, cleared all the crap off my PC (uninstalled programs I no longer use), and did a total virus/malware scan and found 10 threats (including 3 trojans) which I removed. My computer will sometimes randomly crash, sometimes won't let me use some functions of windows 8.

My thoughts:
Potentially a virus that nothing has yet detected? Possibly something to do with my operating system (not sure how that could effect memory usage and other problems though)? Something to do with my RAM (key note: in CPU-Z, DRAM Frequency is stated to be 666.5MHz. I read through another article somewhere and apparently this is half of the true value, implying my actual frequency is 1333MHz?)?

Sorry for all the writing, I just really want to know what's going on with my rig!

basically go to gino and link a ten bag then within the app "out and about" reinstall that and then restart your computer this should do the trick hope this works for you

That AMD GPU is not one of the finest for gaming-----------adequate at best. You might try replacing it with an AMD 8750 or a newer NVIDIA card to loosen that possible bottleneck. Otherwise, try running CCleaner and a GOOD up-to-date Anti-Malware program. That many threats indicates you lack a solid security base for your unit. (Try IOBIT or Symantec for anti-virus and anti-malware-----or Kapersky (a little more expensive). Your CPU and memory are both MORE than adequate for any system today, That CPU/memory fluctuation could be a bad memory chip. Hopefully, you have a 4-slot board, so, remove 2 and reboot. If things look good (with only 1/2 your memory), then replace 1 chip with an "extra" until you've tested all 4. Otherwise, you have a good idea of the culprit.