Depends on which connections you are able to make. I play through my monitor on an HDMI -> DVI cable. It all comes down to what your monitor can support.
Here's the skinny on connections vs quality if you've got more than one port and you can't decide...
Best picture quality to worst:
Oh, and a bit of advice on how to set up your system: the PS3 needs to be set up to run on the new screen BEFORE you connect it to the screen. That means you need to plug it into your old TV and then make the setup changes (that you want to output through HDMI or composite). Then you can plug the PS3 into your monitor and use it as you normally would.
Also, the PS3 has the ability to split audio and video signals and run each to a different location. This is useful because, your screen being a computer monitor, it's unlikely there will be built-in speakers (besides, built-in speakers are almost always of inferior quality). You're going to want to run your audio out the composite wires (or optical audio, should you be lucky enough to afford nice audio equipment) and into your speakers.
Hope this helps.