Ok, so just go by the manufacturer's specifications. Your card is listed at 269mm long, so now just go look up your case's maximum supported GPU length and make sure to pick out a 1080Ti that fits within that limit. If you're not 100% sure that you're looking at the right info, you'll need to list your case model as well so a suitable recommendation can be made, but as was stated above, if you're eyeballing it and you have ~5" of space left, you'll probably be ok with almost any model 1080Ti.
Also, while your PSU will be sufficient for the combination of R5 1600 + 1080Ti at stock configuration, it may be cutting it close if you decide to overclock both components. The 1080Ti alone can (depending on model) pull in 300W+ when overclocked and heavily stressed, and the remaining ~250W capacity for the rest of your system means that the PSU will be working at high load % and therefore running hotter and creating more noise compared to a higher capacity unit. Again, not that it won't work, it just won't be optimal as most PSUs are at peak efficiency between 50-70% load. Just food for thought depending on how hard you're looking to push your system.