I assume you are keeping your desktop resolution at it's native 4K. So if you select the in-game resolution setting to 1080p and disable full screen the game still shows in full screen? Have you tried launch the game(s) at 4K and then exit it, and then go back in and change the resolution to 1080p in the game? The game should hold the last settings you left it in, including the windowed mode. If it triggers full screen with a resolution setting lower than your monitor's native display, then it's got to be how they programmed the game.
Another thought: if you are gaming through the Steam portal, you might be able to toggle the Steam overlay during gameplay (which pauses the game) and that may trigger Windowed mode.