Core i3 + Ati HD 5850 or Core 2 quad + Ati HD 5770, which will be best for 1440x

An AMD system would be much cheaper. What's your total budget for whatever system you're planning?

Well for 1440x900,a HD 5770 is a waste,so if you can change your options,get an
i5 750/HD 5770.
Otherwise,between these 2 since socket 775 is out of production,i would go for the first one.
But like i said,for 1440x900,a 5770 is more than enough,there is no need to get a 5850

What are you using the system for and what components do you already have? For that resolution the 5850 might be overkill so I would recommend your 5770 solution.

Edit: At stock settings the 5770 will also offer you a much better value per dollar so it's definitely a better choice if you're trying to be conscious of how you spend your dollar.

The i3/5850 is the better long term choice. As Maziar pointed out, you have the ability to install an i5-750 or other quad core option later on.

However, at 1440x900 the ATI 5770 is all you'd need. Anything more than that is frankly a waste of money, unless you plan to buy a new monitor soon. The 5770 will do decent at 1920x1080 (1080p) with some games, so it also depends on what you're playing.

I could see doing an i5-750 + 5770 as the better choice. Ultimately, the CPU will perform a lot better, and at 1440x900 the 5770 is more than adequate. The $150 difference between a 5770 and 5850 will easily pay for the upgrade from the i3 to i5-750.