I placed washers over the metalized holes. I heard later that I wasn't supposed to, and simply just screw the screws in the standoffs. Do I need to take the washers off or will I be fine?

For safety reasons remove them, they might shift and cause a short which will probably kill your motherboard. I got lucky, my msi mb survived after i accidentally shorted it

You'll be fine, provided all the standoffs are screwed in tight, and the backplate is even. I've done it both ways: with or without the washers. Just don't overtighten the screws; snug is ok. Sometimes I leave a motherboard hole open when there's no standoff under it. You only need 3-5 screws to secure the motherboard. And of course remove any case standoffs that don't line up with the board holes before mounting.

Paper washers = fine.
Metal washers = bad.