Make sure everything is disconnected first. Check all screws and if you're replacing it with a better model give it a good yank. I've replaced the PSU in a case almost the same (a Compaq which is owned by HP, only difference is the front) and didn't have any problems
In some HP models there is a small tab you press to release a latch holding the PSU in place. Look on the back plate of the case for anything like this.