Yep. If you do that, it shouldn't cause you any problems.
Whenever I've done an install in the past, I've made sure to only have essential hardware in there whilst doing it. It makes for much less confusion and it's easier to configure devices once windows is up and running.
Also, for XP x64, make sure you have a copy of your ethernet drivers on a floppy or a usb drive so you can load them on after you install, because it might not install them automatically for you.