Did you try hot plug?
I mean OS is running
Not yet. The documentation says it will not auto rebuild, but I will try it and let you know.
If this doesn't work, I am going to take rand_79s suggestion and back up the data and rebuild the RAID.