i wouldn't make the upgrade form a gtx 260 unless you went to a 5850 at least, i personally wouldn't recommend the 5770 as other mentioned before is actually slower than 260,
IMO the 5830 performance increase over a 5770 is fairly little, and wouldn't even be that noticeable compared to your current setup
You have a factory OCed GTX 260, so it'll perform better than a stock HD5770. The 5850 might be worth it in some games and might be worth it if you want to run DirectX 11. Performance-wise, it won't be a huge boost but still noticeable in very demanding games.
It also depends on your computer setup, especially the monitor. If you have a monitor with a maximum resolution of 1680x1050, it might not be worth it. Another factor is your CPU. Slower CPUs will cause a bottleneck effect with high-end video cards. You might not see a performance boost in some games because the CPU is limiting your GPU. If you can list your system specs and monitor, it'll be easier to understand what your best choices are.
What about adding another GTX 260 to run in SLI? A 5850 would provide you with generally better performance, but it wouldn't be a big upgrade. The main thing you would get is eyefinity and DX11. A 5770 would actually be a little slower or just as fast as your current card depending on if you have the sp192 or an sp 216 card.