If it's what I'm thinking, you have to kind of squeeze the grommets in the holes to get them in place and removing them is a bitch with damaging them.
Does this sound right? If so...just try your best to squeeze them back out...but I damage them doing this sometimes. It's just a bad design.