It's to do with the way Gmod handles RAM allocation - very badly.
Custom maps require Gmod to allocate a lot of RAM, and constantly read/write the data.
It is notoriously bad for this, and everyone will experience this problem, no matter how good your PC is.
I think there is a way to workaround this problem, but I don't know what it is. Google it, there will be a guide somewhere :P