or im just confusing myself and just ignore it?
This... you're putting yourself through undue hardship by fretting over it. Don't... trust me, it's fine. You'll never touch that shared system memory since the 680 has plenty of VRAM. The shared memory just addressed in case you ever needed it (which you won't).