Shift-Delete work fine but not great for your case. If you can reappear the Recycle Bin on your desktop, Right-click on it and choose "Properties". Then, just choose "Do not move files to ...".
It also shows up under 'my computer' when viewed from explorer. You can empty it there. The desktop icon is just a shortcut to the real recycle bin anyway.