According to the Gigabyte website the MB supports memory up to DDR3 3200:
https://www.gigabyte.com/Motherboard/GA-Z97X-Gaming-5-rev-10#sp
What speed is the current RAM? You may be able to add 2 extra 4Gb modules to bump it up to 16Gb at 'minimum' cost, but if it's at DDR3 1600 ( for example ) moving up to 16Gb of DDR3 2666 will add the memory you're after while helping a little with overall performance.
Re the edit:
OK, good call with the SSD.
Right now there's little real need to go to the expense of swapping out the entire CPU/MB/RAM, maybe, possibly next year, but not now.
If you want to push that nice 144Hz display your best bet will be to change the GPU, but the GTX980 isn't exactly a slow part, and a replacement is going to be in the GTX1070Ti range. With the inevitable Christmas overspend on the way I'll cautiously advise you hold off until a little later when your finances settle down a bit.
Incautiously, and assuming you won't have to sell body parts to fund the upgrade, the GTX1070Ti looks like the best choice-the bigger 1080 isn't that much faster while the '1080Ti is considerably more expensive but check prices very carefully, paying an extra-say-$20 for a GTX1080 would be worthwhile over a GTX1070Ti because it IS a little faster.
Also, consider at least looking into overclocking the current CPU, the system is set up to overclock, and adding a few hundred MHz to the CPU speed can yield some useful overall gains, just be cautious, it is possible to damage the CPU when overclocking, but if you take it carefully and methodically the operation is safe enough.
One point: Do NOT make any voltage changes until you're more familiar with overclocking, it's here that serious damage can be done. Make a mistake elsewhere and most systems will simply reboot automatically several times before booting with the BIOS set to the factory defaults, either that or you'll have to manually reset the BIOS and all should be well, add too much voltage and the CPU can be fried instantly.