Its neither.
720p and 1080p are both 16:9 formats while 1280x1024 is 4:3, so it isnt really applicable.
My guess is that Youtube will letterbox the video, so basically it will have black sides and the footage will be square shaped. You will lose some visual quality when it does this as its basically making the video smaller to fit it in the frame (its taller than 720p).
If you have video editing software, you could crop the footage to 720p, basically cutting off the top and bottom of it and then it will fit natively onto Youtube.
legend001523 :
1080p, but I think doesn't the recording depend on your games' settings?
1920x1080 =/= 1280x1040