You'll be fine, provided all the standoffs are screwed in tight, and the backplate is even. I've done it both ways: with or without the washers. Just don't overtighten the screws; snug is ok. Sometimes I leave a motherboard hole open when there's no standoff under it. You only need 3-5 screws to secure the motherboard. And of course remove any case standoffs that don't line up with the board holes before mounting.