Sorry if this is way late for this discussion, but "bill001g" is correct. The actual WoL frame is a broadcast. You are performing a Unicast IP request to get to the proper subnet, but the encapsulated frame is a broadcast.
There is no "construct" at the end of the path. The WoL frame is constructed at the start, and then wrapped in the packet, with it's IP address to get to the proper subnet. Once it gets to the switch level, the packet has been scraped off the frame, and the WoL broadcast for that subnet is left.
If you enable WoL on the computer, it responds to the call. If WoL isn't enabled, it ignores the broadcast and doesn't respond.
As far as exploits are concerned, I'm not aware of anything at that level. It's a switch to turn on, that's it, there's no "ack", "sync", or any other communication. It's a one way bus to that subnet. I don't mean to make this sound like I'm making light of this, because I'm no expert by any means. If someone kept randomly sending WoL frames to your subnet, it would wake up your machine, but it wouldn't be anymore exposed than it being "On." Once the machine is on, it doesn't attempt to turn on again. I see someone trying to PING ya to death before they use something as obscure as WoL.
As far as your previous statement about a Layer 3 switch, yes you are correct. It absolutely will have an ARP cache in addition to a MAC table. However, it's two devices, which do two different things, wrapped together in the same box, and uses the same CLI or GUI to interact with either component as needed. They still operate separately as far as how they work with the OSI model. Basically instead of having a router sitting on top of a switch and them connected, they are internally connected within the same physical box. They are great because you don't have to login twice, faster communication, some even have auto checks between each side, etc...