Mic the amp- is always better for sound quality.
Buy adapters XLR to RCA, much cheaper than a whole cable!
But, you understand that you can not use the XLR inputs and RCA inputs at the same time? You have to switch between one or the other.
Otherwise, you need a mixer with more inputs, laddie.
Now that I have looked at what you are doing, you need at least 12 input mixer--- 4 vocal mics, three guitar mics, drum mic(s), bass mic, keyboard mic, etc...
the outputs of the 12 channel will feed the inputs of your Tascam.
No, there is no way that a four channel mixer is going to do this job, time to graduate to a real mixer.