In theory, you could do it. But given the hassle, you'd have to seriously consider whether it was worth the effort.
Each computer would need two network adapters (you’re only going to be able to use the maximum number of common connections, which in this case is two). And each computer would need to be multi-homed over IP (e.g., 192.168.1.x for one, 192.168.2.x for the other). Finally, you would segregate the file transfers based on IPs. And yes, obviously Gigabit (1000Mbps) switching would perform better than 10/100Mbps switching provided your drives were capable of Gigabit speeds.
One other word of caution is on order. Many cheap/inexpensive switches do not provide full capacity on their backplane. A consumer-grade, 4-port, 100Mbps switch, which you might *assume* has a 400Mbps backplane, may be considerably less, which ultimately limits the throughput of the switch as a whole. The manufacturer is cutting corners under the assumption the customer is not likely to saturate the switch. That’s why using a high-end switch or inexpensive Gigabit switch, may be necessary, even if you don’t have Gigabit network adapters.