You would have to contact Alienware, to if there is an upgrade option for that laptop. Most of them are not upgradeable, and I dont think that madel is, but check. If you can upgrade I doubt if you could go to a 560m, as usually upgrades of this type are limited to what was the best available at the time.
GTX 260m is a MXM 3.0 type B card like the GTX 560m and GTX 580m but I don't think there is enough power and/or cooling for GTX 580m. GTX 560m may be ok though. Also there is the bios recognition problem. Check with Dell for definite answer.