Going to be tough to find I suspect. If the server and the computer are on a actual switch and are on the same subnet the traffic should go in one port and out the other and not care about anything else. What could be happening is that the application on the server is trying to talk to another machines that does go though the modem. For example asking a login server if your machine has rights to access the application. Your firewall comment makes me think your configuration may actually be more complex than both machines on a single switch.
Best advice I can give is to load a copy of wireshark on the server and capture the traffic when it works and when it does not. You will want to carefully look at the mac addresses it is using to ensure it is not doing something strange like send the traffic to the router rather than directly to the client machines.