Like "Someone" said.. haha... You won't have the ability to NAT, so unless your ISP hands you more than one IP, you won't have more than one device have access at a time. Also, most cheaper Layer 3 switches use static routing, so you can't configure your routing until you know your IP address.
Another issue I thought about when I also thought about taking this route is the gateway IP of your switch typically is also the management IP. This means your switch's management interface will be public on the Internet, depending on what kind of switch you have.
I highly recommend having a proper firewall. I myself have recently built a PFSense firewall that hangs around 4% cpu usage, even when NATing about 1.5gb/s.
And you get pretty graphs