Go with the 1060 3gb + 1700.
I have a 1060 3gb, where it only has 90% of the performance of an actual 1060 6gb, and I also do some video production/rendering and I stream frequently as well. The 3gb version serves me well enough. From my experience running an i7-6700k (at base clock of 4ghz) and a 1060 3gb, CS:GO can be streamed on max settings and output 1080p60 (4500kbps) just fine, in-game fps count is 180-250 (depending on the map). BDO, PAYDAY2, and most other games run on high settings and can output 1080p60 while streaming just fine.
The only games you may have trouble streaming/recording are open world games (usually more graphical complexity, and more CPU reliant), such as GTA V, Watch_Dogs2, Witcher, etc. But hey, even a 1070 struggles with streaming GTA V, WD2 and Witcher in 1080p. So even if you got a 1060 6gb, it will literally make no difference when it comes to open worlds. That's why you should get the better CPU.
Keep in mind, rendering + streaming is more heavily based on the CPU rather than the GPU. Ryzen has overall better performance than Intel when it comes to full utilization while rendering/encoding, but I would still suggest getting the better CPU in your case for maximum effectiveness.
When you are focused on rendering/encoding, don't worry too much about cpu/gpu balance and "I dont want this to bottleneck the other" and such. As long as you can get a GPU that gets the job done with a reasonable quality (1060 3gb does this very well in most games), reach for the best CPU you can get.
If you need further clarification, let me know and I'll try to help!
- Kiri