You have a factory OCed GTX 260, so it'll perform better than a stock HD5770. The 5850 might be worth it in some games and might be worth it if you want to run DirectX 11. Performance-wise, it won't be a huge boost but still noticeable in very demanding games.
It also depends on your computer setup, especially the monitor. If you have a monitor with a maximum resolution of 1680x1050, it might not be worth it. Another factor is your CPU. Slower CPUs will cause a bottleneck effect with high-end video cards. You might not see a performance boost in some games because the CPU is limiting your GPU. If you can list your system specs and monitor, it'll be easier to understand what your best choices are.