The main issue would be if you are using a monitor with a TN panel (most cheap LCDs on the market are TN). If you mount one of them in portrait, the image will look very strange (and bad) from one side, even from a fairly shallow angle. If the monitor is IPS or VA, then no, there's nothing wrong with it. Many of the higher end VA and IPS monitors even have stands that allow you to put them in portrait mode.
I can only assume this monitor is TN. I've held it up portrait to see what it looks like and it's okay. Other than the image looking strange I take it there is not problem with mounting it as portrait then?