X-Fi Forte 7.1 on 2.0 speaker setup [Need advice]

That ought to work fine, just use the headphone/line out connector. If you have a line out connector, seperate from the headphone one use that. The site for the Audioengine 5 speakers says that they are active, i.e they have an amplifier built in, which means you can drive them from a headphone socket.

The reason why it tells you not to connect passive speakers is simply that the headphone amplifier in the sound card is not suited to drive a low impedance load such as a hi-fi speaker directly; basically it can't supply enough current. Headphones ARE usually passive, i.e. simply a small speaker, but they have quite a high resistance compared to a normal hi-fi speaker.

Active speakers solve this problem by including an amplifier, so that your headphone output does not drive the speakers directly. Instead, the input of an active speaker is presents a moderately high resistance which the headphone socket is quite happy to drive.

Wow, thats pretty informative thanks  

but which would be ideal producing the best sound? headphone jack or using the D-SUB 15 pin connector?

I don't think it matters, pick whichever is most conveneient. According to the site for the soundcard:

"3.5mm(1/8 ”) stereo mini-phone jack: Includes an independent Headphone amp and can activate an impedance 16Ω ~ 600Ω headphone or earphone. Also, can be used, when configuring a Stereo system, not using the Analog I/O cable, as an Front Line Output"

I understand that to mean that you could use either one to drive your speakers (since they are active rather than passive), so pick whichever is most convenient, there will be no appreciable difference in sound quality. I personaly would have thought you should use the line out from the D-SUB connector, since this leaves your headphone socket free so that you can plug in headphones without unplugging your speakers.

I would like prefer to not use the D-SUB connector as it would look messier. I am using razer megalodon 7.1 headphone for gaming which is a USB headset.

I am actually very very keen on asus essence stx as its better for music and I use USB headphone for gaming but base on my previous experience on Xonar D2X for my HTPC, it caused Call of duty: World at War, Prototype, FEAR2 to have no sound. Wolverine the video game couldn't even start because of it.

I have already updated to the latest driver for the soundcard.Using back the Onboard soundcard did not have all this problem.
[All those problems faced were on XP SP3]

The soundcard I am about to buy[either forte or STX to replace XtremeMusic] will be used on Vista SP2
Weirdly the X-Fi Forte 7.1 SGD$20 more expensive then Asus Xonar STX from where I live. Shouldn't the forte be cheaper?

Sorry, but I dont know! I have never used a dedicated sound card, personally I have always considered them a waste of money. I have never been dissatisfied with any of the motherboard intergrated soundcards, they work perfectly.

I use the SPIDIF out and connect it up to an external DAC, amplifier and hi-fi speakers. On the rare occasions when I have hooked up via the line out it sounds excellent, to be brutally honest almost indistinguishable from the output of my much better external DAC. I do appreciate though that others may have different needs, better quality ADCs for external inputs and so on.....

Anyway.. if what you want to do is use the headphone socket then use the headphone socket not the line out. They arent going to sound any different.