I will explain why your system blue screens after about three seconds of booting into the windows os.
There are two golden rules when it comes to memory. The first is as I`m sure you know is not to mix memory speeds.
Now even if the memory is provided by the same vendor such as Corsair, the memory speed and latency of the memory chips are always different.
So you have XMS 3 @ 1600Mhz at a latency value of 9-9-9-12 when set to XMP mode.
And Vengence @ 1866Mhz with latency of 9-10-9-27 when set to XMP mode.
The higher the rated memory speed, result in a higher latency speed, the amount of time per clock cycle the memory refreshes.
So if you overclock for example XMS to the speed of the Vengence then the latency values may be so far out it causes error with the memory.
The main reason why also has to do with how the memory is used, if in Dual channel or quad channel memory mode.
Due to the large difference in latency in order to achieve the speed of the vengeance the memory becomes out of sync.
If a memory stick cannot function at a rated Latency/ it`s refresh cycle it results in windows crashing.
One stick of memory is throwing data faster than the slower one and it`s dropping the ball and not catching it.
So data gets corrupted, or lost.
Put simply you would have to make sure that your slowest set of memory,the XMS is placed and seated in the first memory slot of the mobo, or if two sticks of XMS @16000 Mhz the first slots required for dual channel memory mode operation.
Next place the Vengeance and in the bios manually select the speed of the XMS memory. And set it`s latency values.
Test the system with that config, it should be stable.
Simply the XMS cannot keep up or be set to the same latency values of the vengeance at the rated speed of the vengeance. so you have to do a bit of manual tinkering.
By setting the lowest value you improve the situation.
The down side is due to mixed memory speed. You will have to put up with the speed The XMS works at, and not the new stick of Vengeance If you bought it for more ram its not such a loss.
But by doing the way mentioned it should get you out if the Blue screen windows crash problem.
Or it`s buy another set of the vengeance my friend to keep it in sync.