bhattman :
I have i5 2500k processor and MoBo is Gigabyte GA H61M-D2H [possibly Rev 1.0]. It uses DDR3 RAM and has 16GB capacity of 4x4 GB slots. Will Radeon HD 7770 or NVidia GT650 Ti work on it? Can a DDR5 compatible card like those work on my MoBo? Is my MoBo compatible with DDR5 RAM or GFX card? Do I need DDR5 RAM if it works? Currently I use NVidia GT 610 card on PCE v2.0
If the said proposed cards cannot work I am also open to using GT640 DDR3 card if it works. What would be the best card in between GT640 - GT650 range?
All the cards you mention will work.
For HD 7770 and Nvidia 650 Ti you need at least 450W power supply with one 6-pin PCI-Express power connector for it to work properly.
Don't be confused with system memory (RAM) and graphics memory.
The graphics card memory refers to the graphic card itself and you don't need to be worried about compatibility.
If you really want to game with decent performance avoid lower end graphics card like GT 610,620,630 or 640.
Hope this helps.