As has been mentioned, you MUST have a source of "makeup" outside air for the vent fans to work right. We have the same problem, with cold air coming out of the furnace vent. Simple solution, while running the bath or kitchen exhaust fans, crack open the nearest window to the fan. Yes, it will bring in cold air, but it will take the shortest/easiest route to the fan. Doing that should eliminate most, if not all the cold air coming in from the furnace vent. You could also do like the "manufactured homes" do for fireplaces/stoves that are installed. They create an air intake vent directly connected to, or under the fireplace/stove so it doesn't create a draft or starve the stove for air. But, that is a little more involved!