Omg. I am so disappointed and frustrated lol. This whole time it was a single bent pin. And since I kept trying to push the connector in, it bent pretty bad. The pin is curved and bend downwards. This explains why it was never snug.
I guess I can live with how it currently working. It's not touch any other pin and when I plug the connector, everything works fine. Any suggestions on how to fix a pin that is bent downwards? Pencil trick wont work here...I need to get it out and then straighten it.