It depends on the software. A game like Skyrim saves its preferences in a directory along with the save files. Any mods are within the game directory. A game like Dragon Age saves its saved games and DLC seperate from the game directory. A game like Borderlands has its DLC within the game's main directory. A game like Everquest saves your customized UI configuration within the main game directory, as well as any custom UIs, or the maps (back those up if you edit your in-game maps).
There's not a lot of uniformity between games. What you need to back up, where its located, is all a case by case basis. I don't think backing everything up is necessary, but if you measure the time to cherry pick select files, remember exactly where they go, versus just backing up the whole thing if you have the space, you might just opt to back up everything and call it a day.