The sky is the limit if you use some imagination and some reading on the components. Some HPs have a standard ATX board, some don’t. The mix seems to be 50/50. But as long as your board is ATX you can get a cheap Case, PSU, and video card and get some pretty good frame rates.
This is not a recommendation but I got my hands on 2 HP slim line PCs. One is a D530 P4 3.2HT with 2 gigs of ram. For $35 I put an HD3450 in it along with a clean install of Win7 32 and it makes a great HTPC/Netflix PC. It even runs valve games like portal 2 on low. It fits in the rack with a DVD player and Cable box and looks very normal. Guests don’t even know there is a PC there. I use a cheap USB wireless lap keyboard with touch pad.
The other had no model # stickers on it but it is a Pentium Dual Core 3.4 with 4 gigs of ram. I added a low pro Sparkle 9800GT for around $100. It has been going strong now for 2 years on the original 200W PSU! It makes a great HTPC and second gaming PC.