No danger. If it is too large, it may slow your system a little from the increased reading/writing to the drive while the system is using it. I have found that allowing the system to manage its size gives better overall performance.
If you have 2-4GBs of RAM, make it the same size as your RAM. If you have more RAM than that , I wouldn't set it at more than 4GB.