That's right. There will certainly be a difference, but it won't be very much. Clock speed and things like that will make much more of a difference.
check out tom's charts:http://www.tomshardware.com/charts/2012-vga-gpgpu/23-Power-Consumtion-Load,2981.html
If you check the Gainward Geforce GTX 580 Phantom 3gb vs the 1.5 gb you see it makes a difference of 6 watts (297 vs. 291)