So just have them swap some cables when they start work.
Computer is somewhere between the two desks, and has two sets of display/peripheral cables going to it from each desk. When one of them comes in, they just unplug the other guys cables and plug theirs in.
Might have to make use of extensions to make it work and its not automated, but it achieves the goal and its simple.
Wouldn't it be much easier to use a switch rather than swap cables.