any HDD - something cheap with perpendicular recording; SATA is preferred over IDE
any cheap optical drive - SATA again is preferred over IDE
one other thought which i am sure others might want to comment on - for low end systems such as something to run linux on, dell and gateway can put together a low end system for less than you can build it for. i'm not saying you should or should not build it yourself, i'm just saying it might be worth looking at what dell or gateway has to offer for the same level of power and see what their price is on it. if you go that route, i think gateway's customer service dept has a much better reputation than dell's, which seems to have gone downhill in a big way since they outsourced it out of the US. the dell horror stories on the web would be funny if they were not so sad.
lastly, the above parts list likely will overclock quite well if you ever decide you want to learn to OC. (if you know in advance you want to OC it for some reason, there are better ventilated cases than what is listed and an aftermarket cooler such as a Zalman, which is reasonably priced, would serve you better than the stock cooler, but even with the stock cooler you can OC quite a bit).
For a cheap GNU/Linux box I would buy one of the micro atx boards with the video on the board, and one Gig of ram will be all you really need even with Beryl.
Get one of the new Seagate 32mb cache hard drives and stay with an ide optical drive.
I would splurge on a good PSU though, like the PCPower silencer https://shop.pcpower.com/power-supply/silencer-360-atx....
About $350 with shipping included.
I tried to pick products that I know from use have a good reputation and with most items on the list the shipping is free.
When you buy a case look at the fan vent holes, the best type of vent hole pattern is a honey comb shape, less noise. The best size of fans are 12cm, they too make less noise.