Did you check out Crucial to see how much RAM you can put in there?
But you also might have other issues besides RAM. Too many programs running in the background, malware. That could be why the performance is erratic, rather than snaily all the time.
--And drop the registry cleaner. They don't speed up performance and often damage the registry.
--Run MalwareBytes AntMalware and SuperAntiSpyware instead, at least twice a week.
--Drop the Advanced System Care and use either Avira free or Avast Free.
--Firewall--Comodo, Outpost or ZoneAlarm.
The old RAM is expensive because it's old. (LOL) It's not being made anymore, and you're buying old stock.
If you change the mobo + CPU + RAM, then you're running into money that might make it not worthwhile. Can you put a faster CPU into that mobo? I know the charts are somewhere on this site; maybe my colleagues here can point them out.