Are you talking about the hidden "Recycling Bin" folder that you see on each drive?
If so, that's because there is one on each drive!
When you delete a file there are two ways to go about doing it.
Holding Shift+Delete will let you "delete" the file. This doesn't physically erase the file from the disk, it simply marks it as being deleted in the file system. The data is still intact and on the drive until it is overwritten at a later time. This is the same thing that happens when you empty the recycle bin.
The recycle bin acts as a "safe haven" for files that have been erased by the user, but not without further confirmation in order to prevent user error.
On prior versions of Windows there was only one recycle bin location for the entire system and as such moving a file to the recycle bin from a drive other than the same drive as the recycle bin forced the system to copy it to there. On Windows Vista and Windows 7 each drive has its own recycle bin unless it is disabled. These recycle bins are pooled together logically but are physically distinct to each drive. As such, when a file is deleted in explorer it is moved to the recycle bin on the local drive rather than being copied. This speeds things up tremendously (files which are moved from one folder to another on the same physical media do not actually move, only a small change to the filesystem record is made). You can verify the existence of a $RECYCLE.BIN folder on each drive by going to "Folder and Search Options" > "View" and unchecking "Hide protected operating system files".
In order to prevent a drive from having a recycle bin right click on the bin on the desktop as before and select the errant drive and then select "Don't move files to the recycle bin. Remove files immediately when deleted". Normally you shouldn't have to do this with USB sticks but you will have to do it with external hard drives
Alternatively you can just leave it because it will only store files that are deleted on that drive.
I hope that this finally answers your question!