Age of Empires 3 Performance Issue

First, my computer specs are listed in my sig. So, whenever i play AOE3(max settings, 1024x768) it initially runs around 110-130fps, never dropping below 80fps or so. But say when playing a skirmish with 8 total players, later in the game(after 10min maybe) the fps starts to go down, and will steadily do so until reaching about 20fps. Also, im running the most recent version. So my question here is why do the fps go down so much? I mean i have a fast cpu, fast gpu and plenty of RAM, and nothing is overheating or anything. I would think that this computer would run an older game like AOE3 flawlessly. The only thing i can think of is its because the RAM is only running at 500mhz, which isnt providing enough bandwidth for the game. But i dont even know, can anybody be of assistance?

well it's only AI players whom im playing against, so that rules out network lag. so why would my cpu have trouble rendering the game? its by no means slow(3ghz e6600), and AOE3 is a good 2 years old. thanks for the help red_devil!