The full price of the card is more expensive than a 5770 which is newer, supports Direct X 11 and is similar in performance. If it had been store bought I would say refund without a doubt, but if you bought it real cheap then it is probably worth RMAing as you wont get a card as good for the same price.
If he has receipt then get it RMA'd. If not then get your money back and either buy a new one if you can find it at $140 like Number13 says, if not (on newegg the cheapest is $169.99) then I would go with a 5770, it is a good card. DOn't be put off by a few bad experiences - it is a new and budget card, therefore ALOT of people buy it (compared to older cards such as Nvidia 200 series etc) when it works no one is going to post about it, so all you see are those having problems, it may seem like a high number of people, but the ratio to how many bought to how many have issues will be no different to other cards and indeed other hardware
Look in the forums and think about what you are going to do, common sense is an awesome tool, but, in the end, the choice is yours(older card- tried and proven, newer budget card, cheaper is better-maybe)
Yes, ultimately you need to decide. Don't just look at this forum though, you may find the Homebuilt system useful too - A lot of people there know what they are talking about and it is no coincidence that the 5770 is mentioned (admittedly only if people cannot afford the 5850).
Though above all, I would just read proper review sites regarding them. Base your decision on how the card is assuming it performs as it is supposed to, not on how unlucky or lucky thers have been with their cards. there is no big inherant flaw in the design of either cards