The GTX 1080 is fully compatible with your system and would work without a doubt.
However, pairing a 1080 with a 3770 would cause a severe bottleneck.
This means that when you're playing a game, your CPU will max out way before your GPU causing you to get low framerates, even though your GPU can easily run the game with a better framerates.
Buying a 1080 would, therefore, be a complete waste if you are not planning to upgrade your CPU in the future, and even then I would wait for the 11xx series and see what happens.
If you're desperate for a high-performance GPU, then I would check out something like the 1060 6gb
Hope this answers your question
-AlternativeFactsForPresident