Technically, you should leave the case side on to promote correct airflow. But, if you have very poor airflow, you might have to resort to leaving it off. So, this really comes down to case airflow- here is a simple test:
Utilize temp monitoring solutions like HW Monitor, Core Temp or Real Temp. Have them write to a log file to keep track of your temps.
Leave the side of your case closed and run your PC at load using benchmarks such as 3Dmark, PCmark, Intel Burn Test, Prime95, etc. Note the peak temps during these times.
Remove the side of your case and blow a desk or house fan on the highest speed blowing air into the open case. Repeat the same tests as before. Note your temps.
If your temps remain approximately the same (1-3C variance) you have OK airflow. If your temps are significantly lower with the case side open and the large fan working, you have a case airflow issue.