And you'd be wrong. Go for the fast stuff.
Considering the slowest RAM is still thousands of times faster than the alternative in the HDD, maybe only hundreds of times if using a SSD, I'd still go with more RAM. The majority of people, even those posting here, will never be able to tell the difference in speed between the two kinds of RAM without the aid of some sort of benchmarking program. However, you WILL notice a difference between RAM and swap and not need a benchmarking program.