GTX970's low priority memory segment is 4x the speed of system memory. You will still have 3.5GB Vram of priority memory which is more than the 280x and the 970 also performs a lot better. Not many games need 3.5GB vram anyway, will be loosing out on every aspect, power, efficiency, drivers and performance.
Any gpu, taking it beyond it's Vram capacities will see a lot of Vram data swapping and will sure notice it. Its not advised. Actually the 280x Vram overs would be worse since it would be swapping data from system memory vs 970's 500MB low priority memory.
http://www.pcgamer.com/why-nvidias-gtx-970-slows-down-using-more-than-35gb-vram/