Try F8 and see if it will give you a list of boot option. Then choose the usb one and that will set it to the first boot priority.
Finally, The problem actually was not on my keyboard but on the BIOS. I went on local repair shop, and I was suggested to upgrade my bios. Then, the BIOS was upgraded and the problem was fixed.
Anyway thanks a lot for the help.