Old and slow GPU, Probably an older HDD making recording stutter and CPU not helping out.
Maybe 2 drives. One to play and one to capture might help or even an SSD.
Look at the benches for your card in modern games. IT's way low down and in most cases not included because it's too slow/ not relevant.