Hi! That question mark means that the mac can't find the operating system. (Probably because the hard drive was formatted..) Try reading this page. It looks pretty complicated (plus, since your computer is a G4, it probably won't work)
http://hints.macworld.com/article.php?story=20060301112336384
You could also try to hold down 'C' (with the USB stick plugged in). BUT...
The biggest problem you have is that the PowerPC processor which is in the G4 doesn't support booting from a thumb drive. The first macs that supported that were the ones with the intel processors. You may need to try to burn osx 10.5 to a CD or DVD (if the file is bigger that 4 GB, you'll probably need a Dual Layer DVD, which are a little bit more expensive).
Now since your keyboard isn't made by Apple you might not be able to get this work, but a lot of other keyboards DO work.
Here's what I would try.
-Burn OSX to a CD or DVD
-Put the DVD in the G4 (This computer doesn't have an eject button on the outside, so you'll need to get a mac keyboard that has an 'eject' key on it OR get messy and try to get it open by taking the cover of.... might be messy.)
-Turn off the G4
-Hold down 'C' when you turn on the computer.
-That should boot from the disk. And then you can follow the instructions to install it. (During instillation you'll need to open Disk Utility (Utilities>Disk Utility) and check if your drive is the right format (Mac OS extended (Journaled)) and partitioned to the GUID partition scheme.)
Good luck!