Try it with your PC and see if it functions at all. If it works with the PC and not with the Nexus 4, it's probably because the Nexus 4 expects to be connected as the client, not the host. But it does support USB On-the-go, as in this article:
http://www.androidcentral.com/android-advanced-usb-otg-nexus-4 .
So with the right cable it might work.
Brief, probably wrong explanation of USB OTG: A USB connection requires one host device and one (or more for a hub) client device. USB sticks are clients and can't act as hosts. Cell phones were originally clients, you could connect them to your PC and the PC could use them as a mass-storage device. But connecting to clients (your phone and your audio device) results in no activity, because no-one is acting as the host.
A USB OTG device will usually act as a client, but with a proper cable with the correct circuitry to prompt it it will act as the host.
If you want a better explanation, look up USB OTG on Wikipedia.