By default the Delete button does not move files to trash, you need to use Ctrl+Delete which is major annoyance. This is after 3.2 Gnome I believe and still the same up to 3.10. They say it is a feature but I don’t see it as such.
To fix this, you will need to modify the accels of Gnome, with few simple steps:
Run the following in the terminal as your user:
gsettings set org.gnome.desktop.interface can-change-accels true
Now you are able to give items different buttons.
Select a file that you might want to delete (create New Folder for example) and after you have selected it, go to the Edit menu (right-click doesn’t work) and position your mouse over the Move To Trash field. Once the field is in focus, press the Delete button twice. The button combination on the right should be changed from Ctrl+Delete to Delete only.
That is all, now it is enabled. You should disable changing of accels now because you might accidentally do it for other things:
gsettings set org.gnome.desktop.interface can-change-accels false
This was tested on Gnome 3.4.2 on Debian Wheezy.