The graphics card is suppose to read the Extended display identification data (EDID) from the display's PROM/EEPROM.
The EDID includes information such as the manufacturer's name and serial number, product type, phosphor or filter type, timings supported by the display, display size, luminance data and (for digital displays only) pixel mapping data.
The EDID PROM should still be readable by the graphics card even if the display itself is turned off.
It appears the graphics card couldn't re-identify the display after it was disconnected from the primary port and reconnected.
The DVI-I to D-Sub VGA adapters, that you are using, only passes the analog signals component of the DVI-I signals through to the D-Sub 15-pin VGA connector side that your displays use. The graphics card appears to be primarily designed for digitally connected displays.
It seems that the graphics card and the graphics card driver aren't dealing with the disconnecting and reconnecting of analog connected displays properly.
When you reconnected the display on the primary port did you try powering off or unplugging the display's power cord and then reconnecting the power cord and turning it back on?