You must use the N only (or on your router "up to 450Mbps" in the wireless setup page), WPA2 security with AES encryption or you will be limited to 54Mbps. Note that old G devices will not be able to connect when you do this -- the cost of being backward compatible is being slow.
If you are using the computer in a different room from the wireless, unless you have some significant interference issues, use the 2.4GHz band as it has better range than 5GHz. 5GHz is good for nearby devices and in a place where there are lots of other 2.4GHz networks on channels 1, 6, and 11.
How are you checking your wireless speed, with LANTest or timing large file transfers?