1) While I haven't seen the figures on 802.11n yet, all wireless to date has significantly higher overhead than wired does. The rule of thumb is actual throughput = spec'ed bit rate / 3. In an ideal set up (strong signal, no interference, etc.) you might get that up to spec'ed / 2.
Using this, then, your 320Mbps (assuming that is even real) is cut to more like ~100Mbps to ~160Mbps. That's why 802.11n is sometimes called "true" 100Mbps (compared with the 108Mbps technology's "false" 100Mbps!).
2) Wireless is a shared medium, whereas wired is a star topology. This means that if you have 4 wireless client machines all accessing the wireless with a high-bandwidth-demainding task at the same time, each client will see something more like 25 - 40 Mpbs actual throughput.
So anyway.. if I just use one client for the access point, as you say I might get soemthing from 100 to 160... If I just stay next to the AP, so in this best case scenario having a 10/100 port makes the "real 100 Mbps wireless-N" device useless...