Yes, that's the way it's normally done. If using Windows, configure your optical drive to precede your HDD in the BIOS boot order, and then boot from your Windows installation CD. Initialise and format your drive, and then install your OS.
There are plenty of how-tos on the Internet.
BTW, if your used HDD is an IDE/PATA drive, then make sure your motherboard has an IDE port. Otherwise you'll need to buy an add-on card, or a SATA-PATA converter.