If you want a big phone, get the Note 4. If you want something smaller, wait for the M9. There's no reason to get the M8 now. Wait a few months, it'll be worth it.
That said, I have the Note 4 and it's a great phone. Touchwiz is still horrible, but after changing the launcher, it's a good overall experience. Lollipop should be coming to American carriers soon and less bloatware is rumored to be included.
Both are great choices, but they're two very different phones. Weigh the pros and cons of both phones and decide yourself which would be better for you. The main factor is size, but there are other notable differences such as the M9's superior build quality and the Note's stylus that you should take in account.