Looks like they're selling the parts so I guess this must be compatible:http://www.ebay.ca/itm/16GB-2X-8GB-DDR3-12800S-1600MHz-Alienware-M17x-R3-Notebook-Memory-RAM-/120961271695
I also found this thread on the Dell forum:http://forum.notebookreview.com/alienware-m17x/695471-m17xr3-ram-upgrade.html
But I would suggest you ask a Dell sales representative to be 100% sure.
However, I must warn you not to expect a big performance improvement going from 1333Mhz to 1600Mhz, or maybe there's a 2% increase but you're not likely to notice the difference anyways, so you could stick with 1333Mhz and it will get the job done.
But I'm not sure I understand what you want to do exactly, are you going to buy a new 8 GB kit and replace everything? Or just add 1 new stick of 1600Mhz and match it with your previous 1333Mhz sticks? In this case this should work too but I think the BIOS will automatically downgrade the speed to 1333Mhz for every sticks so that they are all in synch.