New 1366×768 (720P as displayed) TV + PS3 Game Image quality

So I bought a new 40" Sony Bariva TV for 650$ on sale yesterday, and I wanted to have a tv that plays the native resolution of most games 720p.

Now I always thought that 720p would be Yx720...but on this tv its Yx768. Now what would be the best option for my ps3 Configuration? Should I force 720p or 1080i or let it choose by itself...

To me I'm just not getting the image quality that I thought I would get, seems almost like AA issues to me. I can't even explain it.

For example in Killzone 2, when I look at the water from a distance the waves and water effects look sort of dotted out (I dunno if that makes sense), in other words it doesn't look like it when I get up close....

Now what I was thinking was to either plug my ps3 in through the VGA cable, or Grab a smaller TV???

What do you guys think?

Also the TV AUTo detects 720p and 1080i, but I can force 1080p....I'm not that great at this TV stuff and its my first HD TV.

Thank You for the help!

P.S
Should it be HDMI or Component for the TV...I kno HDMI is better, but I mean would it make a difference on this TV?