I think the OS "preemptively" swaps some RAM out when a certain "threshold" usage is reached. Moreover, when disabled completely, the system might just crash. Therefore I wouldn't disable it completely unless you have even more RAM. Even I keep a small 512MB swap on my 8GB machine, worst case scenario: it won't be used.