I would say it is safe to ...

I would say it is safe to use in the toilet (why use freshwater for that?) but for gardens it's trickier, and safety depends on what is in the grey water (what was it used for?) and what you are growing (what will it be used for?).

For instance it may be safe to use on trees or a lawn, but not safe for vegetables that you eat raw (particularly if it contains animal wastes of any kind, even from dish-washing).  Also, if it contains a lot of soaps you can cause problems with too much phosphate in water and soil leading to eutrophication or enrichment. Or, it may contain more toxic cleaning products or even fragrances and surfactants , that may be harmful to ingest from food crops, or harmful to soil microorganisms that you rely on for a healthy garden. 

It isn't a 'yes' or 'no' answer, but water conservation is always a good thing. I would use it to flush, and use things like the water used to cook vegetables, grains, etc. be used for the garden (as long as there is no salt in the water) as it is full of good nutrients, with nothing harmful.