The simplest way is to find out how much heat is produced by irf540 mosfet. Then you know how much heat is to be transferred. Now you can find a heatsink on the market that can do the job. If you want to design the heatsink yourself, then you must decide if you will use a fan or no fan. Then the material must be selected. Then heat transfer calculations come into the picture. You must know the maximum ambient temperature, heat transfer coefficient between the heatsink and the ambient substance, mostly air, the surface area of the heat sink, etc.. You must also consider the aging and surface degeneration of the heatsink material and clogging of the fins.