I am a big user of baby shampoo as well... I don't really like the idea of using chemicals on my brushes since I have pretty sensitive skin.
I also wash whatever brush was used that day. I don't like to think about "schmutz" on my brushes and me rubbing that all over my face for a week or two until I clean them.