Problem solved: Do not use a download accelerator!
I'm using Flashget, which starts automatically when I want to download something
but it doesn't seem to work when downloading drivers.
When I turned it off, it downloaded the driver (the full 100MB) just fine and I had
no problems installing it!
I think the NVidia site actually mentions that somewhere...