I think you're asking the wrong question. You obviously have enough estrogen receptors to develop female sexual characterists, ask any trans-sexual, they just dump a load of hormones into themselves & they develop breasts.
The question should be how ingested (not injected) hormones work & how, if any they differ from phyto-estrogen that are ingested.
I believe no one is claiming that milk from animals affects you're sexual aspect (making a male more female), milk is loaded with female hormones & yet ingesting them does not seem to affect the male body - & people have drunk a lot of milk!!! I've not got the same library of research that is was in my old computer, but here's a bit that mentions the large amount of estrogens in modern milk ( http://www.ncbi.nlm.nih.gov/pubmed/1612 ... stractplus
OK, so we've got to accept that ingested estrogens either do not promote, or have very little effect on the male physiology; this or milk has some 'special', undiscovered property that stops this estrogen being utilised.
So, what about phyto-estrogens? These are chemicals that have a similar chemical structure to estrogen, but are not the same substance. There are several types of research out there at the moment, from what I can see. Basically one type separates out tons of the phyto-estrogens & force-feed some poor creature massive amounts, this will of course give negative results, try that with separated estrogen from cows milk & you’d get an even more pronounced response! Most research done using soya & not separated phyto-estrogens give the phyto-estrogens an anti-estrogen type effect, that is they block any receptors that are active & work much less effectively than a true estrogen would.
Now, that doesn’t mean you should go on a soya binge, but if you treat soya as just another food, then you will not get any negative effect (unless you have allergies to soya etc).
If you are concerned about estrogen-type substances in your food, then steer clear of animal milk, meat from female animals & injecting yourself with estrogen (or too much testosterone as it will convert & you’ll be wearing bitch-tits!). Eating a normal amount of soya products aren’t really a worry.